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Findings (cont'd) 


Objective 

We inspected the F-35 Lightning II 
Program (F-35 Program) at Lockheed 
Martin, Fort Worth, Texas, for conformity 
to the contractually required Aerospace 
Standard (AS)9100, "Quality Management 
Systems - Requirements for Aviation, 

Space and Defense Organizations." 

We also evaluated corrective actions 
taken in response to nonconformities, 
findings, and recommendations 
identified in DoD Inspector General (IG) 
Report No. DODIG-2013-140, 

"Quality Assurance Assessment of 
the F-35 Lightning II Program," 

September 30, 2013, to determine whether 
the actions taken were appropriate. 

Findings 

The F-35 Program generally conformed to 
requirements and showed improvement in 
quality management system performance 
since our previous evaluation; however, 
challenges still remain, as evidenced by 
57 nonconformities to the AS9100 Standard 
and 4 opportunities for improvement. 


B. ensure that all system level requirements and 
capabilities were realized and verified, 

C. create an independent quality assurance organization, 
establish its roles and responsibilities, and ensure it 
was adequately staffed to perform effective oversight 
for the F-35 Program, 

D. ensure that Lockheed Martin was taking necessary 
steps to reduce the assembly defect rate in order to 
meet the full rate production goals, 

F. ensure that Lockheed Martin's software quality 
management processes were performed sufficiently 
to prevent software defects, and 

G. ensure that Lockheed Martin flows down all 
contractual requirements to its subcontractors, 
evaluates deliverables for contract compliance, and 
allows minor nonconformances to be approved only 
by the proper authority. 

In addition, the Defense Contract Management Agency (DCMA) 
did not: 

E. escalate unresolved Corrective Action Requests (CARs) 
to the next higher level as appropriate and required 
by its policy, for effective resolution. 

Recommendations 


The Joint Program Office (JPO) did not: We recommend that the Joint Program Office: 


A. ensure the program made sufficiënt 
progress toward full compliance 
with Public Law 108-136, "National 
Defense Authorization Act for 
fiscal year 2004," Section 802, 
"Quality control in procurement of 
aviation Critical Safety Items (CSIs) 
and related services," and the 
Joint Service CSI Instruction 
(SECNAVINST 4140.2), "Management 
of Aviation Critical Safety Items," 


A.l. ensure that the F-35 CSI Program is compliant with 
Public Law 108-136, Section 802, "Quality control in 
procurement of aviation CSIs and related services," 
and the Joint Service CSI Instruction, "Management 
of Aviation Critical Safety Items"; 

A.2. conduct periodic CSI Program evaluations of 
Lockheed Martin and its suppliers to ensure 
compliance with public law and the Joint Service 
CSI Instruction; 


Visit us on the web ot www.dodig.mil 
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Recommendations (cont'd) 

B.l. clearly define contractual criteria for the 

acceptance of all future and fielded aircraft to 
ensure that aircraft capabilities are verified; 

B. 2. ensure that all 21 system-level requirements 

that may not be met, in addition to the risks 
associated with the failure to meet these 
requirements, are documented, tracked, 
and mitigated using the established risk 
management process; 

C. realign the quality assurance organization 
to report directly to the Program 
Executive Officer, define the organization's 
roles and responsibilities, and staff the 
organization appropriately; 

D. l. ensure that Lockheed Martin implements 

quality improvement initiatives to reduce 
the assembly defect rate to meet full rate 
production goals; 

D.2. coordinate with DCMA to implement an 

effective root cause analysis and corrective 
action process in order to reduce assembly 
defect rate; 

F. work with Lockheed Martin to ensure software 
quality management Systems are improved; 
metrics should be reported on a periodic basis 
(for example, monthly) to evaluate process 
improvement; and 

G. l. ensure that all minor nonconformances are 

evaluated and approved only by DCMA. 


We recommend that the DCMA: 

E.l. review all unresolved CARs and escalate those 
that meet the criteria established in DCMA 
policies and instructions, 

E.2. assess all CARs that were not properly elevated 
and assess any impact on the product, and 

G.2. ensure that Lockheed Martin flows down 

the appropriate technical requirements to its 
subcontractors and receives and evaluates 
contract deliverables within the required 
time frames. 

Management Comments and 
Our Response 

On January 20, 2015, JPO and DCMA provided 
comments on our findings and recommendations. 

The Joint Program Office agreed with 
six recommendations and partially agreed 
with three recommendations. 

JPO partially agreed with the recommendation to track 
21 system-level requirements, which it acknowledged 
will not be met. However, JPO does not consider the 
21 system-level requirements as risks and did not 
agree to track them in its formal risk management 
process. We disagree with this approach because 
a final determination of performance has not been 
made and failure to track the risks in the formal risk 
management process prevents an identification of the 
plans necessary for closure. 
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Management Comments and Our Response (cont'd) 

JPO partially agreed with the recommendation to 
work with DCMA to implement an effective root cause 
analysis and corrective action process to reduce 
assembly defects to meet full-rate production goals. 
However, JPO stated that no additional changes to 
corrective action processes were necessary. We 
disagree with JPO's response because additional quality 
initiatives are required to meet full-rate production 
goals and DCMA's involvement is necessary to ensure 
objectives are met. Our recommendation was for JPO 
to take actions to identify and correct the cause of the 
program's inability to reduce defect rates to support 
full-rate production. 


JPO also partially agreed with our recommendation 
to ensure all minor nonconformances are evaluated 
and approved only by DCMA. The actions that JPO is 
planning to meet the intent of the recommendation. 

DCMA agreed with all three of our recommendations. 
The actions that DCMA is planning to take meet the 
intent of the recommendations. 

Please see the Recommendations Table on the 
following page. 
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Recommendations Table 


Management 

Recommendations 
Requiring Comment 

No Additional 
Comments Required 

Joint Program Office 

A.l, B.l, B.2, C, D.l, D.2, and F 

A.2 and G.l 

Defense Contract Management Agency 


E.l, E.2, and G.2 


Please provide comments by April 13, 2015. 
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INSPECTOR GENERAL 

DEPARTMENT OF DEFENSE 

4800 MARK CENTER DRIVE 

ALEXANDRIA, VIRGINIA 22350-1500 


March 11, 2015 

MEMORANDUM FOR PROGRAM EXECUTIVE OFFICE JOINT STRIKE FIGHTER 

DIRECTOR, DEFENSE CONTRACT MANAGEMENT AGENCY 

SUBJECT: F-35 Lightning II Program Quality Assurance and Corrective Action 
Evaluation (Report No. DODIG-2015-092) 

We are providing this report for review and comment. The DoD Office of Inspector 
General (OIG) conducted an evaluation of the F-35 Lightning II Program (F-35 Program) 
for conformity to the contractually required Aerospace Standard (ASJ9100, "Quality 
Management Systems - Requirements for Aviation, Space and Defense Organizations,” 
and to determine whether the program took appropriate actions on findings and 
recommendations identified in the DoD Inspector General Report No. DODIG-2013-140, 
"Quality Assurance Assessment of the F-35 Lightning II Program," September 30, 2013. 

We conducted this evaluation in accordance with the Council of Inspectors General on 
Integrity and Efficiency, "Quality Standards for Inspection and Evaluation." 

Our evaluation determined that the F-35 Program generally conformed to requirements 
and showed improvement in its quality management system performance with 
respect to our previous evaluation; however, challenges still remain, as evidenced by 
57 nonconformities to the AS9100 Standard and 4 opportunities for improvement. 

We considered management comments on the draft of this report when preparing the 
final report. DoD Instruction 7650.03 requires that recommendations be resolved 
promptly. Comments from the Defense Contract Management Agency were responsive, 
and do not require additional comments. Comments from the F-35 Joint Program Office 
were not fully responsive; therefore, we request further comments on Recommendations 
A.l, B.l, B.2, C, D.l, D.2, and F by April 13, 2015. 

Please provide comments that conform to the requirements fo r DoD Instruction 765 0.03. 
If possible, please send a PDF file containing your comments 

Copies of your comments must have the actual signature of the authorizing official 
for your organization. We cannot accept the /Signed/ symbol in place of the actual 
signature. If you arrange to send classified comments electronically, you must send 
them over the SECRET Internet Protocol Router NetWork (SIPRNET). 
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We appreciate the courtesies extended to the staff. Please direct questions to 




Jeputy Inspector General 
Policy and Oversight 


cc: Under Secretary of Defense for Acquisition, Technology and Logistics 

Assistant Secretary of the Air Force (Financial Management and Comptroller) 

Naval Inspector General 

Auditor General, Department of the Navy 
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Introduction 


Objectives 

Our objective was to evaluate and verify the implementation of the corrective 
actions taken by the F-35 Joint Program Office (JPO), Defense Contract Management 
Agency (DCMA), and the prime contractor—Lockheed Martin Aeronautics 
Company (Lockheed Martin), Fort Worth, Texas, in response to the findings 
identified in DoD Inspector General (IG) Report No. DODIG-2013-140, "Quality 
Assurance Assessment of the F-35 Lightning II Program," September 30, 2013. 

As part of this project, we also reevaluated Lockheed Martin’s implementation of 
the contractually required Aerospace Standard (AS)9100, "Quality Management 
Systems - Requirements for Aviation, Space and Defense Organizations," 
contractual quality assurance clauses, internal quality assurance processes and 
procedures, and public law. In addition, we evaluated JPO- and DCMA-assigned 
functions and responsibilities related to oversight and monitoring of 
F-35 contractors. 

Background 

The F-35 Lightning II Program (F-35 Program) is a joint, multiservice, 
multinational acquisition to develop and field a next-generation strike fighter 
aircraft for the Navy, Air Force, Marine Corps, and numerous international 
partners and foreign military customers. The F-35 has three variants; model 
F-35A—Conventional Takeoff and Landing (CTOL), F-35B—Short Takeoff and 
Vertical Landing (STOVL), and F-35C—Carrier-Suitable Variant (CV). In addition 
to hundreds of other suppliers worldwide, Lockheed Martin has two principal 
subcontractors/suppliers, Northrop Grumman Aerospace Systems (Northrop 
Grumman) and BAE Systems (BAE). 

The F-35 Program entered the System Development and Demonstration (SDD) 
phase of the acquisition lifecycle in October 2001; however, after exceeding 
critical cost growth thresholds established by statute—a condition known as a 
critical 1 Nunn-McCurdy breach—the original Milestone B decision was rescinded in 
June 2010 and was not re-approved until March 2012, after program restructuring. 
As identified in the F-35 acquisition strategy, the F-35 Program uses development, 
production, and sustainment activities concurrently with 11 separate low-rate 
initial production (LRIP) lot deliveries. Each LRIP lot represents an increasing level 
of maturity and additional system capabilities are incrementally delivered as the 
program approaches full-rate production. 

1 Section 2433, title 10, United States Code defines a critical "Nunn-McCurdy" unit cost breach as an increase of at least 
25 percent over the program acquisition unit cost (PAUC) or average procurement unit cost for the program (APUC) or 
subprogram as shown in the current baseline estimate or an increase of at least 50 percent over the PAUC or APUC for the 
program or subprogram as shown in the original baseline estimate. 
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As of September 2014, Lockheed Martin delivered a total of 106 aircraft 
including the majority of LRIP lot 5 aircraft and was planning to begin delivery of 
LRIP lot 6 aircraft. Lockheed Martin plans to deliver LRIP lots 6, 7, and 8 aircraft 
with Block 3I 2 software capabilities. LRIP lots 9 and beyond will be delivered with 
additional software block updates to enable full warfighting capabilities. 

Summary of Previous Evaluation Results 

From mid-2012 until early 2013, the DoD Office of Inspector General (OIG) 
performed a quality management system evaluation of the F-35 Program 
at Lockheed Martin and its major subcontractors. 3 This evaluation 
resulted in 363 nonconformities 4 to the AS9100 Standard. Eighty-seven of 
these nonconformities were written against Lockheed Martin; 63 were 
documented at Lockheed Martin, Fort Worth, Texas and 24 were documented 
at its subcontractor sites. The remaining 276 nonconformities documented 
during our previous evaluation were written against Northrop Grumman; 

BAE; Honeywell Aerospace; L-3 Display Systems; and United Technologies 
Corporation, Aerospace Systems (UTAS), and were not included in this 
evaluation. All nonconformities were categorized as major, minor, or an 
opportunity for improvement (OFI), in accordance with AS9101, "Quality 
Management Systems - Audit Requirements for Aviation, Space, and Defense 
Organizations." Of the 63 nonconformities that were directed to Lockheed Martin 
at its Fort Worth, Texas, facility, 23 were categorized as major and 40 were 
categorized as minor. These nonconformities were further categorized by the 
applicable top-level AS9100 clauses as shown in Figure 1. 


2 Initial training and warfighting capabilities were included in software blocks prior to Block 31. Block 31 includes third 
generation helmet and night Vision capabilities. Future software block upgrades will include full warfighting capabilities. 

3 DoD IG Report No. DODIG-2013-140, "Quality Assurance Assessment of the F-35 Lightning II Program." 

4 AS9101D defines a major nonconformity as "a non-fulfillment of a requirement which is likely to result in the failure of 
the quality management system or reduce its ability to assure controlled processes or compliant products/services" and 
a minor nonconformity as "a non-fulfillment of a requirement which is not likely to result in the failure of the quality 
management system or reduce its ability to assure controlled processes or compliant products/services." 
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Figure 1. Nonconformities Written Against Lockheed Martin at its Fort Worth, Texas, 
Facility Documented in Report No. DODIG-2013-140 



Evaluation Criteria 

We performed our evaluation based on the AS9100C Standard, which the contractor 
is required to comply with in accordance with the F-35 contracts. Additionally, we 
evaluated the F-35 Program’s compliance with applicable statutory and regulatory 
requirements to include the Federal Acquisition Regulation; Defense Federal 
Acquisition Regulation Supplement; DoD Instruction Interim 5000.02, "Operation 
of the Defense Acquisition System," enclosure 1; and DoD Manual 4140.01, "DoD 
Supply Chain Materiel Management Procedures: Materiel Sourcing,” volume 3. 
Furthermore, we evaluated the F-35 Program’s implementation of aviation 
critical safety item (CSI) requirements. Defense Federal Acquisition Regulation 
Supplement (DFARS) Subpart 209.270, "Aviation and Ship Critical Safety Items” 
defines an aviation CSI as "a part, an assembly, installation equipment, launch 
equipment, recovery equipment, or support equipment for an aircraft or aviation 
weapons system" that contains a characteristic such that any failure, malfunction, 
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or absence of which could cause a catastrophic or critical failure resulting in the 
loss of life, permanent disability or major injury, loss of a system, or significant 
equipment damage. As part of this evaluation, we performed a quality assurance 
inspection for two weeks at Lockheed Martin, Fort Worth, Texas from late 
September 2014 through early October 2014. See Appendix A for more details 
on the criteria of this evaluation. 

Quality Assurance Evaluation Process 

To evaluate JPO’s management of the F-35 quality assurance program and 
Lockheed Martin’s implementation of corrective actions from the previous 
DoD IG report (Report No. DODIG-2013-140), we performed a quality assurance 
inspection for two weeks at Lockheed Martin, Fort Worth, Texas from late 
September 2014 through early October 2014. The results enabled us to evaluate 
the F-35 Program’s conformity to the contractually required AS9100C quality 
management system Standard, contractual quality assurance clauses, and internal 
Lockheed Martin quality assurance processes and procedures. It also allowed 
us to evaluate Lockheed Martin’s progress on its quality management system 
performance. Additionally, we evaluated the corrective actions taken in response 
to the nonconformities discussed in the previous DoD IG evaluation report and 
verified whether the nonconformities were adequately resolved or on Schedule 
for resolution. 

We established teams of engineering and subject matter experts who evaluated the 
program based on the AS9100C quality management system Standard. The subject 
matter expert teams consisted of quality assurance engineers, trained and certified 
in AS9100, who had an average of 17 years of quality assurance audit experience. 
Additionally, we included a team that evaluated the Lockheed Martin’s aviation 
CSI Program. 

Classification and Categorization of Nonconformities 

As defined by the AS9101 Standard, a major nonconformity is a nonfulfillment of a 
requirement that is likely to result in the failure of the quality management system 
or reduce its ability to ensure controlled processes or compliant products/services. 
A minor nonconformity is a nonfulfillment of a requirement that is not likely 
to result in the failure of the quality management system or reduce its ability 
to ensure controlled processes or compliant products or services. An OFI is an 
industry best practice where a specific requirement does not exist. This report 
focuses on the major nonconformities documented during this evaluation. 
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Organization of the Report 

The remainder of this report is organized in three sections, the first of which 
discusses the results of our AS9100 inspection of Lockheed Martin conducted 
from September 22 to October 3, 2014 , the second documents the results of 
our evaluation of the corrective actions taken in response to our previous 
report (DODIG-2013-140), and the third section provides a summary of the overall 
performance of the F-35 program quality management processes by comparing the 
results of this evaluation to the results of our previous evaluation. We conclude 
the report with a discussion of our overall findings and recommendations. 
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Current F-35 Program Quality 
Management Performance 

Results of AS9100 Conformance Inspection at 
Lockheed Martin, Fort Worth, Texas 

Our inspection at Lockheed Martin, Fort Worth, Texas from late September 2014 
through early October 2014 resulted in 61 nonconformities that identified 
weaknesses in the F-35 Program's implementation of the AS9100 quality 
management system. Of these, 52 were written against Lockheed Martin and were 
categorized as 24 major, 26 minor, and 2 OFIs. Four nonconformities were written 
against JPO: two major and two OFIs. In addition, five were written against DCMA: 
one major and four minor. 


Figure 2. Lockheed Martin Nonconformities Found During the Evaluation 
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Lack of Commitment to Quality Management Objectives 

Management Commitment (5.1) 

In accordance with the F-35 Program Quality Management Plan, a 90-percent 
reduction of the assembly defect rate is necessary by full-rate production in 
order to meet the goal of producing one aircraft per day. 5 At the time of our 
inspection, LRIP lot 5 should have demonstrated a 60-percent reduction in defect 
rate; however, the required progress toward this goal could not be determined 
because—in accordance with conflicting information contained in the quality 
management plan—Lockheed Martin was tracking the number of nonconforming 
material records rather than the number of assembly defects. Nonconformance 
data for LRIP lots 1 through 5 demonstrated only a 23-percent reduction of 
nonconforming material records. The lack of ability to accurately determine and 
reduce assembly defects in accordance with defined program objectives may 
negatively impact the full-rate production delivery Schedule. 

Insufficiënt Software Quality Management Rigor 

Competence, Training, and Awareness (6.2.2) 

We identified a systemic problem with Lockheed Martin’s software engineer 
training program. lts Learning Management System and Training Requirements 
Matrix only captured training completion status and did not include 
F-35 project-specific training plans or training requirements for software 
developers. The development of software intensive systems, such as the 
F-35 Program, requires software development integrated product team members 
involved in the design, implementation, and testing of critical software to 
receive specific training applicable to their assigned tasks. Without a software 
development training program, software defects could increase and possibly 
degrade product performance, reliability, and maintainability. 

Design and Development Verification (7.3.5) 

Lockheed Martin was not conducting functional configuration audits (FCAs) and 
physical configuration audits (PCAs) of the Mission System and Vehicle System 
computer software configuration items (CSCIs). The intent of an FCA is to verify 
that a CSCI (or a hardware configuration item [HWCI]) meets the requirements 
documented in the associated specifications by examination of the test plans, 
procedures, and data. A PCA is formal evaluation of the as-built configuration 
of a HWCI or a CSCI against its technical documentation to establish the 


5 The F-35 Program Quality Management Plan defines the assembly defect rate reduction goal with respect to the 
SDD baseline and prescribes a 10-percent reduction per LRIP lot in order to achieve a 90-percent reduction by full 
rate production. 
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product baseline (as-built versus as-designed). Lockheed Martin required all 
of its software suppliers to verify their CSCIs by performing FCAs and PCAs; 
however, Lockheed Martin did not perform FCAs/PCAs for internally developed 
Mission System and Vehicle System CSCIs. Furthermore, the lack of complete 
software requirements definition of lower level software units such as Fire Control 
and Navigation hampered Lockheed Martin’s ability to perform FCAs and PCAs. 

The absence of FCA and PCA requirements for internally developed software 
could result in software that is not verified and validated, leading to aircraft 
performance risks. 

Vehfication of Purchased Product (7.4.3) 

Lockheed Martin received software from suppliers without ensuring that known 
deficiencies were categorized and defined in accordance with its software 
development plan (SDP). For example, Northrop Grumman delivered Integrated 
Communications, Navigation, and Identification (ICNI) software to Lockheed Martin 
with 1 major, 12 minor, and 92 known open and uncategorized deficiencies. 
However, neither the ICNI SDP nor the F-35 Air System SDP defines software defect 
categories or severity as major and minor. Lockheed Martin instead categorizes 
software deficiencies as Severity 1 through 5 and does not allow the delivery of 
flight software with open deficiencies of Severity 1 or 2. Lockheed Martin did 
not determine the system performance impact and severity of these deficiencies. 

If software deficiencies are not categorized by severity as defined in the 
F-35 Air System SDP, then software could be released with open high-severity 
defects, which could result in performance degradation or even catastrophic 
failure of aircraft. 

Preventive Action (8.5.3) 

We found numerous examples of software problem anomaly reports (SPARs) 
identified with Severity 2 problems 6 that did not address root cause analysis or 
effective preventive action. The root cause identification and corrective action 
processes are critical for product improvement and reliability growth and the 
lack thereof may also result in design and quality escapes that impact product 
performance, reliability, and maintainability. 


Joint Strike Fighter Program: Air System Software Quantitative Management Plan, Document Number 2RXP00003, 
Revision D, defines Severity 2 software defects as "[a] software defect that adversely affects the accomplishment of an 
operational or mission essential capability and no work-around solution is known." 
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Unmanaged Performance Specification Risks 

Risk Management (7.1.2) 

JPO and Lockheed Martin jointly identified 21 out of 462 system-level requirements 
contained within the Joint Strike Fighter (JSF) Contract Specification (JCS) 
that were unlikely to be achieved by the end of the SDD phase of the program. 
Examples of these 21 requirements included: Reliability and Maintainability, 
Maneuverability, Payload Requirements, Ballistic Vulnerability, and CTOL Internal 
Gun Accuracy. However, Lockheed Martin did not consider these 21 requirements 
as risks and did not handle them in accordance with its risk management process. 
Fulfillment of these requirements may not be achieved if they are not appropriately 
tracked and managed as risks. 

Weaknesses in Subcontract Management 

Purchasing Process (7.4.1) 

JPO requires Lockheed Martin to be AS9100 compliant. Lockheed Martin’s internal 
procurement process flows down AS9100 or appropriate quality management 
systems to its suppliers based on product complexity and criticality. However, 
three of Lockheed Martin’s approved suppliers were not AS9100 certified, as 
required by Lockheed Martin’s internal supplier quality requirements. Therefore, 
those suppliers might not have the necessary quality management system 
processes in place to ensure product reliability. 

DCMA is the only JPO-delegated authority to approve minor nonconformances. 
However, Lockheed Martin procurement requirements allowed minor 
nonconformances to be dispositioned and approved by suppliers who 
provide "seller-designed” items. Allowing subtier suppliers to disposition 
and approve minor nonconformances impedes the Government’s ability to 
fulfill its responsibility to manage the risks associated with the acceptance of 
nonconforming products. 

Purchasing Identification (7.4.2) 

Lockheed Martin’s procurement process did not ensure that its purchase orders 
identified the specific revisions of technical documentation required to produce, 
inspect, and test the item being purchased. We also found that when purchase 
orders were issued, Lockheed Martin relied on its suppliers to retrieve the most 
up-to-date technical requirements from their online technical data system rather 
than notifying the supplier when technical data had changed. These practices 
could result in products that are not compliant with the applicable technical 
requirements in order to achieve mission capabilities. 
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Vehfication of Purchased Product (7.4.3) 

Our inspection identified that Lockheed Martin was not using its escalation process 
for delinquent supplier-generated data items to ensure its subcontractors delivered 
the required data items on time. Hundreds of delinquent supplier data items 
existed with an average delinquency of about 230 days. Many data items were 
more than 2 years delinquent, with one item exceeding 1,200 days. Examples of 
such delinquent data items included manufacturing plans, acceptance test plans 
and procedures, and diminishing manufacturing sources redesign and technology 
refresh plans. Many of these subcontractor data items contain information that 
must be included in Lockheed Martin’s contractually required deliverables to JPO. 
Lockheed Martin has an escalation process to address delinquent subcontract data 
item submittals that includes withholding funds; however, no action was taken to 
use the escalation process for any of its suppliers. Manufacturing plans, acceptance 
test plans and procedures and other documentation discussed previously should 
be evaluated by the appropriate subject matter experts for adequacy. Failure to 
evaluate such deliverables could result in deficiënt manufacturing or validation 
processes and ultimately impact product performance and reliability. 

Weaknesses in Production and Assembly 

Our inspection of Lockheed Martin’s manufacturing and assembly areas resulted 
in the identification of deficiencies in process discipline, process planning, process 
proofing, tooi control and verification, and product protection. 

Control of Production and Service Provision (7.5.1) 

We identified a lack of process discipline in the manufacturing and assembly area. 
In one instance, multiple fasteners on the Radio Frequency Support Electronics 
assembly strut were not torqued in accordance with the applicable specifications. 
The manufacturing documentation, referred to as operational cards and work 
instructions, indicated that both the operator and quality inspector had verified 
that the torque operation was completed. However, we observed that the fasteners 
were loose and rotated freely. It was also identified that a mechanic in the final 
assembly area used sealant on an arresting gear assembly fasteners after the 
sealant’s useful life had expired. 

In another instance, we found that operational card or planning did not specify 
the required torque values for the mounting of a cabling connector to its bracket 
on the frame of the forward fuselage. Torque requirements for these fasteners 
were prescribed in a Lockheed Martin specification referenced by the assembly’s 
engineering drawing. As a result, the mechanic installed the fasteners without the 
use of a torque wrench and was not aware of the torque requirement. 
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Lockheed Martin did not adequately perform process proofing of the automated 
drilling operations on the CTOL wing. lts proofing process released automated 
drilling software for production that was not programmed correctly to achieve 
full penetration of the wing edge in accordance with the applicable specifications. 
Verification operations failed to identify the improperly drilled holes before the 
product left the work center. These holes were identified at the next work station, 
and the corrective action taken was to rework the holes to meet the design 
requirements using a hand drill process. 

We found no evidence of a formal process or management system to track 
the status and condition of critical lift fixtures. As a result, required periodic 
inspection and load testing of critical lift fixtures were not performed as a system, 
to include: lifting fixture, lifting strap, rings, clevises, and pins. We also noted that 
a production tooi with an expired periodic inspection/verification tag was used in 
the wing assembly area. 

We documented four minor nonconformities related to Lockheed Martin’s 
measuring equipment control and calibration process that indicate the existence 
of a systemic problem. Examples of deficiencies include: calibration labeling, 
maintenance certification, and the availability of tools for use while outside the 
maintenance certification period. 

Lack of adequate control in the area of manufacturing and assembly operations 
through the use of disciplined implementation, planning, process proofing, and tooi 
control and verification could increase defect rates, prevent achievement of the 
planned production delivery Schedule, and result in negative impacts to product 
performance and reliability. 

Preservation of Product (7.5.5) 

During our inspection of the production areas, we found one employee walking 
on an aircraft without the required shoe covers to protect aircraft surfaces and 
also walking on surface areas designated as "no-step” areas. Walking on no-step 
areas could stress, fatigue, or damage aircraft structures. In addition, required 
protective material was not used to minimize the risk of damage to the aircraft 
surfaces by automatic lifting work stands and platforms. 

We also identified discipline issues with Foreign Object Debris (FOD) protection 
and housekeeping. FOD was detected at two FOD control areas in the final 
assembly area. FOD has continued to be a systemic discipline issue since our last 
evaluation, as evidenced by 41 Corrective Action Requests (CARs) issued by DCMA 
for FOD violations during 2013 and 2014. Additionally, Lockheed Martin did not 
adhere to housekeeping policies and practices for cleaning all work stations and 
auto drill centers of metal chips. 
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There is increased risk of latent defects and damage to the aircraft, if 
Lockheed Martin does not ensure procedures are followed and product 
protection measures are taken. 

Government Quality Assurance Oversight Performance 

Performance of DCMA 

We evaluated DCMA’s oversight of F-35 production and identified that DCMA 
was not following its process for escalating CARs. DCMA’s policy is to escalate 
contractor CARs to the next higher level when repetitive CARs have been issued, 
when rejected Corrective Action Plans (CAPs) are not resubmitted within the 
10-day requirement, or when corrective action validation is rejected. We found 
instances when DCMA failed to escalate CARs which exemplified each of these 
cases. The reoccurrence of FOD-related problems highlighted the ineffectiveness 
of DCMA’s corrective action oversight. DCMA issued a CAR to Lockheed Martin in 
response to FOD control deficiencies documented during our previous evaluation. 
This CAR was subsequently closed based on the validation of completed corrective 
actions. During 2013 and 2014, since our previous evaluation, DCMA issued 
39 Level II CARs for instances of FOD violations without escalating them to 
Level III. Failure to escalate CARs to the next higher level when appropriate 
undermines the effectiveness of the corrective action process. In addition, DCMA 
and JPO leadership may not be aware of problems until they adversely impact the 
cost, Schedule, and performance of the program. 

Performance of the Joint Program Office 

As previously stated in report section Unmanned Performance Specification 
Risks, Risk Management (7.1.2), Lockheed Martin has been unable to achieve 
21 system-level capability requirements. In accordance with the H-30, "Future 
Specification Changes," SDD contract clause issued in May 2013, JPO intend to 
change the 21 high-risk system-level JCS requirements to values mutually agreed 
upon by JPO and Lockheed Martin based on additional testing and analysis which 
is yet to be performed. As a result, delivered and future aircraft may not satisfy all 
system capability requirements and it is unclear to what extent these capabilities 
will be achieved by the completion of the SDD phase. 
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Results of Corrective Action Evaluation 


As a part of this effort, we evaluated the actions taken by Lockheed Martin, JPO, 
and DCMA in response to the findings documented during the previous evaluation 
to determine whether the root causes were identified and the corrective actions 
were effective. 

A total of 87 nonconformities were documented against Lockheed Martin during 
the previous evaluation; 63 were documented at Fort Worth, Texas, and 24 were 
documented at its subcontractor sites. 

JPO stated that Lockheed Martin implemented corrective measures to resolve these 
nonconformities and considered 82 of them to be closed. However, we determined 
that only 76 of the 87 nonconformities were closed and we recommended 
further actions for the 11 remaining nonconformities; 5 were due to ongoing 
corrective action and 6 were due to inadequate corrective action. The criteria we 
used to determine whether previous nonconformities were still open included: 
ongoing corrective action efforts, ineffective corrective action to prevent finding 
reoccurrence, administrative closure of findings prior to assessment of corrective 
action effectiveness, and inadequate documentation of corrective actions taken. 

To ensure that all nonconformities requiring further action from the previous 
evaluation are properly tracked to closure, we documented each item as a new 
nonconformity during this evaluation. We consider the following issues still open. 

Ongoing Corrective Action 
Critical Safety Items 

During our previous evaluation, we documented four CSI nonconformities related 
to the lack of a CSI Program. These nonconformities were considered open by 
JPO, DCMA, and Lockheed Martin due to incomplete implementation of corrective 
action. Since our previous evaluation, Lockheed Martin worked with JPO and 
DCMA to develop a plan to establish an F-35 CSI management program. Status 
and sufficiency of the plan is discussed in the Government Progress on Corrective 
Actions section of this report. 

Capability Confirmation Criteria 

We also documented a nonconformity regarding the lack of capability confirmation 
criteria for the acceptance of LRIP aircraft during our previous evaluation. This 
nonconformity was still considered open by JPO, DCMA, and Lockheed Martin 
due to incomplete implementation of the corrective action. Since our previous 
evaluation, Lockheed Martin worked with JPO and DCMA to develop a plan 
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to establish capability confirmation criteria. On LRIP contracts, the program 
developed a Configuration and Capability Description Document (CCDD) to 
identify specific configuration and representative/key capability attributes, such 
as warfighting capabilities available to the pilot and the maintainer, which would 
be present in a respective LRIP lot. However, at the time of this evaluation, 
Lockheed Martin and JPO were at a contractual impasse and were unable to 
implement the associated corrective actions. JPO and Lockheed Martin had 
no further plan or path forward to establish verifiable capability confirmation 
criteria for the acceptance of LRIP aircraft. The lack of clearly defined capability 
confirmation criteria in the CCDD resulted in acceptance of aircraft whose 
capabilities may not have been adequately verified and validated. 

Ineffective Corrective Action 

Foreign Object Debris 

JPO, DCMA, and Lockheed Martin considered the systemic FOD issues to be 
resolved and the associated corrective actions closed. However, we identified 
that the corrective actions did not effectively prevent reoccurrence of FOD issues. 

In response to our recommendation from our previous report, Lockheed Martin 
implemented an 8-step CAP. Internal Lockheed Martin production assessment 
scorecard data for 2014 and DCMA Level II CARs for 2013 and 2014 indicated that 
the steps taken in the CAP had not been effective in preventing reoccurrence of 
FOD in manufacturing areas. An ineffective FOD control program could lead to 
damaged aircraft, degraded performance, catastrophic failure, loss of aircraft, and 
loss of life. 

First Article Inspection 

First article inspections are required to ensure that manufacturing processes 
are validated properly to reduce variability and nonconformances in delivered 
products. Our previous evaluation at L-3 Display Systems (L-3) identified that 
Lockheed Martin accepted hardware from L-3 for LRIP lots 1 through 5 without 
the performance of first article inspections. JPO, DCMA, and Lockheed Martin 
considered this issue to be resolved and administratively closed the CAP in 
December 2012; however, the documentation provided with the CAP stated that the 
sustainability or effectiveness of corrective actions taken could not be confirmed 
until the third or fourth quarter of 2013. We found no evidence that verification 
and validation efforts were performed for these corrective actions by JPO, DCMA, 
or Lockheed Martin. 
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Variance Approval Process 

A nonconformity regarding Lockheed Martin’s approval of variances that waived 
performance requirements for the electronics and display units was documented 
at L-3 during our previous evaluation. The variances did not contain documented 
corrective actions that would prevent the need for repeated variance requests. 

As part of the resolution of this deficiency, Lockheed Martin implemented a 

3- step CAP and although JPO, DCMA, and Lockheed Martin considered this issue 
to be resolved and closed, our evaluation identified that Lockheed Martin’s actions 
to resolve this nonconformity were ineffective. Specifically, Lockheed Martin’s 
procedure and records indicated that integrated product team members were 
allowed to extend the variances to an unlimited number of units and extend 
commitment dates associated with corrective action completion without 
management approval. Because variance requests typically impact product 
performance—as documented in this case—the acceptance of product variances 
without adequate control, oversight, and corrective action could result in degraded 
aircraft capabilities. 

Failure Reporting and Corrective Action 

During our previous evaluation we documented that Lockheed Martin did not 
require UTAS to deliver Failure Reporting and Corrective Action System (FRACAS) 
data to determine the reliability of the products delivered. JPO, DCMA, and 
Lockheed Martin considered this issue to be resolved and closed. However, 
we identified that the corrective actions to resolve this issue did not prevent 
reoccurrence. Since our last evaluation, Lockheed Martin implemented a 

4- step CAP to address identified deficiencies. UTAS delivered the FRACAS data 
for LRIP lot 5 after funding issues were resolved; however, funding constraints 
prohibited the delivery of FRACAS data for LRIP lots 6 and 7. Lockheed Martin 
cannot ensure that the F-35 meets all product reliability requirements if its 
engineering staff does not review and analyze subcontractor FRACAS data. 

Annual Audit of Software Suppliers 

We documented a nonconformity during our previous evaluation relating 
to Lockheed Martin's oversight of software suppliers. JPO, DCMA, and 
Lockheed Martin considered this nonconformity to be resolved and closed; 
however, our evaluation identified that the corrective action was inadequate. 
Lockheed Martin did not audit its software suppliers annually in accordance 
with the applicable internal operating instruction. We evaluated the records 
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of five software suppliers and noted that Lockheed Martin had not conducted 
required annual audits for any of the suppliers since we identified this issue during 
our last evaluation. Because much of the aircraft capability is reliant upon robust 
software, adequate control of software suppliers is necessary to minimize the risk 
of accepting defective software that can degrade product performance, reliability, 
and maintainability. 

Calibration of Measuring Equipment 

During our previous evaluation, we documented that Lockheed Martin granted 
temporary extensions of calibration due dates for measuring equipment whose 
calibration certifications had already expired. The associated CAP identified that 
the root cause of this nonconformity was the failure of the equipment owners 
to submit calibration extension requests prior to expiration with adequate 
justification. As a result, Lockheed Martin removed the requirement to request 
extension prior to calibration expiration. Based on this corrective action, JPO, 
DCMA, and Lockheed Martin considered the nonconformity resolved and closed; 
however, this could result in the increased risk of using un-calibrated tools, 
potentially leading to the manufacture of nonconforming product. We determined 
that the root cause analysis and corrective action should be reevaluated to prevent 
reoccurrence of the problem. 

Inadequate Objective Evidence 

Lockheed Martin was not always able to provide objective evidence to demonstrate 
that corrective actions taken to resolve nonconformities identified in our previous 
evaluation were completed. Lockheed Martin uses its Integrated Corrective 
Action (ICA) database to capture internal CARs, CAPs, and objective evidence to 
support CAR closure and CAP validation. The ICA database did not contain all 
objective evidence for numerous CAPs and their associated DCMA CARs to support 
verification of corrective actions. Closure of nonconformities without necessary 
objective evidence to support validation of CAP actions can lead to reoccurrence of 
the nonconformities. 

Government Progress on Corrective Actions 

We also evaluated DCMA’s and JPO’s actions taken in response to nonconformities 
documented against them during our previous evaluation to determine if root 
causes were identified and corrective actions were effective. 
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Defense Contract Management Agency 

During the previous evaluation, we documented a total of six nonconformities 
against DCMA offices. These offices included: DCMA Fort Worth, Texas; 

DCMA Northern Europe, Samlesbury, United Kingdom; DCMA Atlanta, Georgia; and 
DCMA Cleveland, Ohio. We identified deficiencies in the delegation of authority, 
minor variance approval, material review board actions, issuance and execution of 
letters of delegation, and delivery of supplier reporting data between DCMA offices. 
Based on the information presented by DCMA, we determined that closure of these 
six findings was substantiated. 

Joint Program Office 

Previously Documented Nonconformities 

We documented eight nonconformities against JPO during our previous evaluation. 
These nonconformities related to the CSI Program, documentation of F-35 risk 
management activities, F-35 diminishing manufacturing sources and materiel 
shortages (DMSMS) management and planning, and Block 2A test readiness review 
criteria. JPO considered three nonconformities closed; however, it did not provide 
any objective evidence for corrective actions taken in response to any of the 
eight nonconformities. 

Status ofCritical Safety Item Program Implementation 

We documented in our previous evaluation that JPO and Lockheed Martin did 
not comply with Public Law 108-136, Section 802; DFARS Subpart 209.270; 
and the Joint Service CSI Instruction, "Management of Aviation Critical Safety 
Items." Since then, Lockheed Martin created an extensive CSI list and DCMA 
added the requirement for government source inspection for all CSIs on the list. 
Lockheed Martin also drafted a CSI Program Plan for submittal to JPO, which 
is expected to be included in a modification to the SDD contract to implement 
the CSI requirement. The draft CSI Program Plan erroneously describes the 
F-35 Program as "a legacy program with respect to CSI," and unfortunately, 
implementation of this plan would lead to less than full compliance with 
CSI requirements. At the time of our evaluation, JPO’s plan was to contractually 
require implementation of a CSI program by February 2016. We determined that 
the draft CSI Program Plan, dated May 21, 2014, does not meet all the Joint Service 
CSI Instruction requirements and Lockheed Martin’s production integrity program 
did not cover all identified CSIs and CSI requirements. 
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During this evaluation, we identified that Lockheed Martin’s CSI Program Plan 
was still noncompliant with some of the deficiencies that we identified in our 
previous report. For example, the F-35 CSI Program Plan contained a section 
describing the Identification and documentation of critical characteristics. Based 
on that section, it appeared that Lockheed Martin concluded that existing quality 
and integrity programs were sufficiënt. Furthermore, the plan did not require 
CSI critical characteristics to be defined. It also did not ensure CSI-specific 
training for personnel performing work on CSIs or put the necessary processes in 
place when operations affect critical characteristics. The CSI Program Plan still 
did not address limitations on material review board authority for nonconforming 
CSI assemblies and/or components or include scheduled internal and external 
CSI audits. 

Status of Independent Quality Organization Establishment 
In our previous report, we recommended that JPO establish an independent 
quality assurance organization. The intent of the recommendation was to 
improve the quality organization’s authority and independence. JPO did initiate 
the establishment of a quality organization and assigned a Quality Lead; however, 
the organization reports to the Director of Production rather than directly to 
the F-35 Program Executive Officer (PEO). The F-35 Program also had not yet 
adequately staffed its quality organization to provide effective oversight, nor had it 
ensured roles and responsibilities of the organization were developed and followed 
to effectively impact the program. JPO’s August 8, 2014, response to our previous 
report stated that the estimated date to have the quality organization developed 
and staffed would be 12 to 24 months from May 2014. 

Status of Capability Confirmation Criteria Implementation 
During this evaluation, we found that the capability confirmation criteria necessary 
to verify and validate LRIP aircraft capabilities during aircraft acceptance had not 
been defined in the CCDD, as documented in our previous evaluation. As a result, 
the acceptance of F-35 aircraft depended on the professional judgment of the 
individual JPO subject matter expert who reviewed the objective evidence provided 
to confirm the capabilities of the aircraft. 

Since our previous evaluation, JPO worked with Lockheed Martin and DCMA to 
develop a plan to establish capability confirmation criteria for aircraft acceptance; 
however, at the time of this evaluation, JPO and Lockheed Martin were at a 
contractual impasse and, as a result, were unable to implement the associated 
corrective action. JPO and Lockheed Martin had no plan or path forward to 
establish capability confirmation criteria for the acceptance of aircraft. 
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Summary of the Evaluation 

Based on our evaluation, we determined that the F-35 Program has made progress; 
however, some challenges still remain. 

Although the quantity of major nonconformities documented during this evaluation 
remained relatively the same in comparison to our previous evaluation, the overall 
number of nonconformities decreased (see Table 1). We further categorized the 
nonconformities by major AS9100 section and compared the results with those 
of our previous evaluation (see Figures 3 and 4). Lockheed Martin’s quality 
management system performance improved in the Human Resources, Planning of 
Product Realization, Design and Development, and Documentation Requirements 
focus areas of AS9100 compliance. We also noted improvement in the Production 
and Service Provision focus area; however, the abundance of major nonconformities 
in this area is still a cause for concern. Among the major nonconformities 
documented in this area, we observed the reoccurrence of a systemic FOD problem, 
indicating ineffective corrective actions. Although the overall trend demonstrated 
improvements in the majority of areas, the nonconformities related to the 
Purchasing and Improvement focus areas of AS9100 compliance increased. 


Table 1. Comparison of Major and Minor Nonconformities Directed to Lockheed Martin 
from the previous report and this evaluation 



Major 

Minor 

Total 

Previous Evaluation* 

23 

40 

63 

Current Evaluation 

24 

26 

50 


* DoD IG Report No. DODIG-2013-140, "Quality Assurance Assessment of the F-35 Lightning II Program," 
documented 70 nonconformities at Lockheed Martin, Fort Worth, 63 of which were directed to 
Lockheed Martin and 7 of which were directed to JPO. 
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Figure 3. Comparison of Major Nonconformities Directed to Lockheed Martin 
(Previous Evaluation and Current Evaluation) 


No. of Nonconformities 



AS9100 Clause 
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Figure 4. Comparison of Minor Nonconformities Directed to Lockheed Martin 
(Previous Evaluation and Current Evaluation) 



DODIG-2015-092 | 21 











Summary of the Evaluation 


We also noted that the F-35 Program made progress with respect to its corrective 
actions in response to nonconformities documented during our previous evaluation. 
In response to the 87 nonconformities documented against Lockheed Martin 
that we identified in our previous report, Lockheed Martin developed and 
implemented corrective action plans and considered 82 of them resolved and 
closed. Lockheed Martin identified and documented root cause, corrective action 
plans, and provided adequate evidence to support closure of the majority of the 
nonconformities documented against them. However, we determined that only 
76 of these 87 nonconformities were closed and we recommended further actions 
for the 11 remaining nonconformities. JPO’s lack of effective oversight to ensure 
timely and efficiënt resolution of deficiencies identified in our previous report is 
demonstrated by the following: 

• limited commitment to a fully functional and independent quality 
oversight organization, 

• lack of ability to produce objective evidence for the closure of 
previous findings and nonconformities, 

• failure to comply with public law in fulfilling the 
CSI Program requirements, 

• lack of progress in defining the acceptance criteria for each LRIP lot, 

• lack of adequate commitment to ensure that the program fulfills all 
system level requirements, and 

• lack of leadership to ensure defect rate reduction necessary to meet the 
planned full-rate production delivery goals. 

In addition, while DCMA has demonstrated improvement, they are not following the 
established DCMA CAR escalation process to manage Lockheed Martin’s corrective 
action activities appropriately. 

Based on our quality management system inspection and evaluation of corrective 
actions, we documented the following systemic findings. 
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Finding A 

Insufficiënt Progress Had Been Made Toward 
Implementation of the Critical Safety Item Program 

The F-35 Program had not made sufficiënt progress toward full compliance with 
Public Law 108-136, Section 802, "Quality control in procurement of aviation 
CSIs and related services,” and the Joint Service CSI Instruction, "Management 
of Aviation Critical Safety Items,” by February 2016. This was caused by the 
lack of CSI requirements on the F-35 contracts, as required by public law and 
the Joint Service CSI Instruction. This lack of compliance with CSI public law 
and implementing instructions may impose an increased safety of flight risk on 
delivered aircraft. 


Discussion 

As identified in our previous report JPO had not flowed down a contractual 
requirement to Lockheed Martin to implement a CSI Program in accordance 
with Public Law 108-136, Section 802 and the Joint Service CSI Instruction. 

The CSI public law has been imposed since 2003. 

JPO and Lockheed Martin have taken initial steps to establish a CSI Program; 
however, the program was still not fully compliant with applicable laws and 
regulations. JPO was planning to contractually require implementation of a 
CSI Program by February 2016. However, the current draft of the CSI Program 
Plan prepared by Lockheed Martin did not sufficiently address requirements 
described in the Joint Service CSI Instruction including the need to define critical 
characteristics for all CSIs and the proper limitations and delegations for the 
material review board authority for nonconforming CSI critical characteristics. 
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Management Comments on the Finding and 
Our Response 

Summaries of management comments on the finding and our response are in 
Appendix C. 

Recommendations, Management Comments, 
and Our Response 

Recommendation A 

We recommend that the F-35 Joint Program Office: 

1. Ensure that the F-35 CSI Program is compliant with Public Law 108-136, 
Section 802, "Quality control in procurement of aviation CSls and related 
services,” and the Joint Service CSI Instruction, "Management of Aviation 
Critical Safety Items." 

F-35 Joint Program Office Comments 

F-35 Joint Program Office agreed and stated: 

• Contract actions for both SDD and LRIP that address the Joint Service 
CSI Instruction, "Management of Aviation CSls" are in work or in place to 
implement the full F-35 CSI Program 

o SDD RFP for CSI NRE [non-recurring engineering] efforts to initiate 
F-35 CSI Program released Oct 2013 and resulted in a LM delivered 
Draft CSI Program Plan coordinated with F-35 JPO for approval of 
the CSI implementation approach and go ahead for delivery of the 
SDD proposal expected; go ahead date for proposal development 
expected 2QFY15 

• After the SDD CSI NRE efforts are completed, the full scope 
of the F-35 CSI program will be included in future production 
and sustainment contracts. 

o LRIP Production contracts (LRIP 1-8) currently cover explicit 
CSI tasks for 'limited' compliance for several CSI 'gaps' from the 
Joint Service CSI Instruction, "Management of Aviation CSls." 

• Maintain / update the DOE [Director of Engineering] 
approved CSI list for configuration control 

• Use existing recurring processes: Material Review (MR), 
Variance, Item Acceptance, New Suppliers, Process Metrics, 
and Minor & Major Change processes for CSls 

• Provide CSI notification to affected suppliers via 
contracts letter 

• Provide notification if any identified non-conformances 
results in a safety impact 
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• Report MR actions & QARs [Quality Assurance Reports] 
associated with CSI once per month 

• Use existing disposal processes to destroy scrapped CSIs 
that render part nonfunctional 

• DCMA-LM FW [Fort Worth] Integrated CSI Surveillance Strategy developed 
Jan 2014 and covers CSI Delegations to ensure appropriate DCMA process 
surveillance of CSI parts, and imposed Government Source Inspection (GSI) 
on all parts that have been identified as CSIs per the Joint Service 
CSI Instruction. 

Our Response 

The comments from JPO partially meet the intent of our recommendation. 

Based on JPO's response, the F-35 program appears to be making progress in 
establishing a CSI program. However, the comments do not include a timeline for 
the implementation of the “full" F-35 CSI Program. We request that JPO provide 
a timeline for the implementation of the “full" F-35 CSI Program and update the 
current draft CSI Program to fully comply with applicable laws and regulations. 

2. Conduct periodic CSI Program evaluations of Lockheed Martin and its 
suppliers to ensure compliance with public law and the Joint Service 
CSI Instruction. 

F-35 Joint Program Office Comments 

F-35 Joint Program Office agreed and stated: 

F-35 JPO & LM [Lockheed Martin] conducted a CSI TIM [Technical Interchange 
Meeting] in Nov 2014 following completion of the DoD IG Quality Assurance 
& Corrective Action Evaluation to address DoD IG “Opportunity For 
Improvement” comments issued for CSIs and CSI Program Plan. 

F-35 JPO, DCMA & LM are conducting bi-weekly telecoms and periodic 
TIMs on-site at LM until SDD contract agreement is reached to implement 
the full scope CSI Program infrastructure and processes. 

After F-35 full scope CSI implementation is underway, the F-35 JPO will 
transition to a monitoring phase with periodic reviews, which will leverage or 
combine with other JPO/DCMA activities (joint audits, Production Readiness 
Reviews, quality reviews, metrics, etcetera). 


Our Response 

The comments from JPO meet the intent of our recommendation. No further 
comments are required. 
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Finding B 

Oversight of Product Development and Realization of 
Requirements Was Inadequate 

JPO was not adequately managing the F-35 Program to ensure that all system-level 
requirements and capabilities defined in the JSF JCS were realized and verified. 

As a result, delivered and future aircraft may not satisfy all system capability 
requirements and it is unclear to what extent these capabilities will be achieved 
by the completion of the SDD phase. 


Discussion 

The JCS document identifies more than 400 system-level requirements that 
define performance characteristics. JPO and Lockheed Martin jointly identified 
21 system-level requirements contained within the JCS that were unlikely to 
be achieved by the end of the SDD phase of the program. However, JPO and 
Lockheed Martin did not consider these 21 requirements as risks and did not 
handle them in accordance with their risk management process. 

In addition, JPO still has not defined objective and measureable aircraft acceptance 
criteria for each LRIP lot. The confirmation criteria defined in the CCDD did not 
include clearly measurable and verifiable acceptance criteria that were clearly 
traceable back to the JCS. This was documented in our previous evaluation and 
continues to be a challenge. 

Management Comments on the Finding 
and Our Response 

Summaries of management comments on the finding and our response are in 
Appendix C. 

Recommendations, Management Comments, 
and Our Response 

Recommendation B 

We recommend that the F-35 Joint Program Office: 

1. Clearly define contractual criteria for the acceptance of all future and 
fielded aircraft to ensure that aircraft capabilities are verified. 
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F-35 Joint Program Office Comments 

F-35 Joint Program Office agreed and stated: 

F-35 JPO is progressively refining acceptance criteria with each LRIP lot. 

The use of CCDDs will continue to be used for LRIP 7 and LRIP 8. These CCDD 
items require positive verification activity to occur, often with dedicated flight 
test evaluations; and they provide a means to ensure the weapons system 
meets warfighter expectation for that production lot. However, the use of 
these documents does not negate the responsibility to document any known 
JCS non-compliances at the time of aircraft acceptance. This will continue to 
occur as it has occurred in the past. As SDD progresses F-35 JPO does expect 
more qualification evidence to be available which will allow for more robust 
documentation of JCS non-compliance at the time of aircraft acceptance. This 
will be done in addition to requiring positive verification for all CCDD items. 

Beginning with LRIP 9, the next lot to be negotiated, F-35 JPO will require full 
JCS compliance. F-35 JPO will assess LRIP 9+ product acceptance against the 
SDD specification. For future full-rate production contracts, there is planned a 
separate production specification that will leverage the final SDD specification. 


Our Response 

The comments from JPO do not meet the intent of our recommendation. JPO has 
not identified the actions that it will take for verification and validation of aircraft 
already delivered or still in production for LRIP lots 1 through 8. JPO needs to 
ensure all aircraft delivered are tested and verified using clear and measurable 
capability criteria. We request that JPO provide further comments in response to 
the final report. 

2. Ensure that all 21 system-level requirements that the F-35 Program has 
acknowledged may not be met, in addition to the risks associated with 
the failure to meet these requirements, are documented, tracked, and 
mitigated using the established risk management process. 

F-35 Joint Program Office Comments 

F-35 Joint Program Office partially agreed and stated: 

F-35 JPO sees merit in tracking to closure the contractually defined 
21 system level requirements that may not be fully met. F-35 JPO agrees 
the high-level visibility of a formal tracking Action Item to periodically 
status these requirements to closure will ensure resolution of them in an 
expeditious manner. 

However, F-35 JPO DISAGREES with the recommendation to track these 
21 system level requirements through the risk management process. These 
21 requirements are not risks - they are a subset of requirements where the 
program has jointly acknowledged that the level of specified performance 
will not be met. Conversely, the purpose of risk management is to highlight 
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and take proactive actions to avert a potentially bad outcome, such as a 
specification miss. Within the F-35 risk management process, risks are 
managed via a series of funded mitigation steps and technical gates to 
eliminate or reduce the likelihood and/or consequence of each risk. With 
these 21 requirements, F-35 JPO has already acknowledged the specification 
misses and has a contractual agreement to spend no more money to improve 
the design - using the cost as an independent variable principle for the 
requirements. The only remaining program resources associated with these 
21 requirements is for gathering qualification evidence to verify the capability 
of the design as is. When qualifications are complete, F-35 JPO will reset 
the specification requirements to the achieved performance and negotiate 
consideration for each of the shortfalls. 

A closure tracking / status Action Item for these 21 system level requirements 
would regularly assess progress toward reconciliation and closure of the 
specification to realized performance: qualification actions, coordination of 
verification evidence, draft specification change request engineering review, 
Configuration Steering Board approval, Change Control Board approval 
and contract negotiations / contract modifications. However, for these 
21 system level requirements, neither risk mitigations nor design changes are 
funded within the remaining scope of the System Design and Development 
contract. As an aside, several of the 21 system level requirements are 
separately tracked as Program Technical Performance Measures. 


Our Response 

The comments from JPO do not meet the intent of our recommendation. Although 
JPO stated that the requirements will not be met, the undetermined extent to which 
the 21 system-level requirements will not be met poses a risk to the F-35 Program. 
These 21 system-level requirements should be tracked as program-level risks 
either until the requirements have been satisfied or relief has been granted by the 
appropriate authority. We request that JPO provide further comments in response 
to the final report. 
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Finding C 

The Quality Assurance Organization Was Not 
Independent and Not Adequately Staffed 

JPO had not defined its quality assurance organization’s roles and responsibilities 
and had not provided adequate staffing to perform effective oversight of the 
F-35 Program. Also, the quality assurance organization reports to the Director 
of Production rather than directly to the F-35 PEO. The JPO quality assurance 
function did not have the necessary independence to implement the quality 
management mission for development, production, and sustainment of the system. 
A quality assurance organization that lacks independence or the inherent authority 
to enforce quality management requirements may not effectively mitigate cost, 
schedule, and performance risks to the F-35 Program. 


Discussion 

Our previous evaluation recommended that JPO establish an independent quality 
assurance organization. JPO initiated the establishment of a quality organization, 
assigned a Quality Lead, and plans to further develop and staff the organization. 
But, the organization reports to the Director of Production rather than directly 
to the F-35 PEO. Because the desire to meet production schedules can often 
prevent adequate or sufficiënt root cause analysis and corrective action, a quality 
assurance organization that is independent and reports to the PEO ensures greater 
influence in program decisions. The current organizational structure may limit 
the quality assurance organization’s ability to objectively report program quality 
management performance. 

Management Comments on the Finding 
and Our Response 

Summaries of management comments on the finding and our response are in 
Appendix C. 
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Recommendations, Management Comments, 
and Our Response 

Recommendation C 

We recommend that the F-35 Joint Program Office realign the quality assurance 
organization to report directly to the PEO, define the organization's roles and 
responsibilities, and staff the organization appropriately. 

F-35 Joint Program Office Comments 

F-35 Joint Program Office agreed and stated: 

Efforts to establish an independent quality organization are on track. 

As documented in F-35 JPO's August 2014 response and reiterated during 
presentations and discussions with DODIG team members, a quality 
organization will be in place 12-24 months from the May 2014 decision. 


Our Response 

The comments from JPO do not meet the intent of our recommendation. 

JPO has proposed inconsistent approaches. JPO’s response letter addressing 
the recommendation in our previous evaluation to establish an independent 
quality organization stated, "[t]he F-35 JPO will establish an independent quality 
organization reporting to the Program Executive Officer.” However, during this 
evaluation, JPO presented a quality organization that has a direct reporting 
line to the Director of Production, which we do not consider to be independent. 

In addition, it is not clear from JPO’s response whether the quality organization 
will be fully functional 12 to 24 months from the referenced May 2014 decision. 
Timely establishment of the quality organization is essential to provide necessary 
oversight, monitor the program performance, and take proactive measures to 
prevent significant risks to the program. We recommend that JPO provide an 
update to the Schedule and milestones associated with the development of the 
quality organization, including a current and proposed quality organization 
structure. We request that JPO provide further comments in response to the 
final report. 
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Finding D 

Reduction of the Assembly Defect Rate 
Was Inadequate 

JPO and DCMA management were not committed to ensuring that assembly defect 
rate objectives and goals were realized. Lack of ability to reduce defects and the 
scrap, rework, and repair rate in accordance with defined program objectives will 
negatively impact program cost and Schedule. 


Discussion 

JPO was not ensuring that Lockheed Martin was fulfilling its commitment to 
a 90-percent assembly defect rate reduction per aircraft to meet the full-rate 
production goal of completing one aircraft per day. We were not able to determine 
Lockheed Martin’s progress in meeting its commitment to a 90-percent reduction in 
the assembly defect rate because it was not tracking its progress using the correct 
data for SDD or LRIP lot deliveries. In accordance with conflicting information 
contained in the quality management plan, Lockheed Martin was tracking the 
number of nonconforming material records rather than the number of assembly 
defects. Lockheed Martin demonstrated a 23-percent reduction in nonconforming 
material records compared to the 60-percent defect rate reduction that should 
have been realized after LRIP lot 5. In addition, JPO and DCMA did not ensure that 
corrective actions for product nonconformances were adequately implemented to 
reduce the assembly defect rate. At the time of our evaluation, we were not aware 
of any initiatives or value engineering change proposals specifically aimed at the 
reduction of the assembly defect rate in accordance with the program quality 
management plan goals. 

Management Comments on the Finding 
and Our Response 

Summaries of management comments on the finding and our response are in 
Appendix C. 
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Recommendations, Management Comments, 
and Our Response 

Recommendation D 

We recommend that the F-35 Joint Program Office: 

1. Ensure that Lockheed Martin implements quality improvement initiatives 
to reduce the assembly defect rate to meet full-rate production goals. 

F-35 Joint Program Office Comments 

F-35 Joint Program Office agreed and stated: 

As presented, Lockheed Martin has initiated several initiatives to reduce 
nonconformances of all types both within their facilities and throughout their 
supply chains. The F-35 JPO is committed to ensuring quality objectives 
and goals are realized, therefore monitors progress closely with monthly 
and quarterly quality metrics reviews and Production Readiness Reviews. 
Additionally F-35 JPO production operations and quality subject matter 
experts attend Systems Engineering Technical Reviews to assure designs 
are producible and encompass quality objectives. Additional improvement 
incentives are being realized through the program's Blueprint for Affordability 
which provides a cost share between the contractor and government to fund 
the improvements, while not solely dedicated to quality and producibility 
improvements the majority of projects achieve savings through defect 
reduction by focusing on producibility. F-35 JSPO [Joint Strike Fighter 
Program Office] is satisfied with the methodology LM is using to monitor SRR 
in lieu of a ratio to production hours. Additional metrics are being evaluated 
by F-35 JPO, DCMA and the contractors independently and jointly to continue 
driving discovery of nonconformances at their source (eliminate escapes] 
as well as facilitating correction. The mutually agreed to quality goals are 
part of the Joint Program Commitments, which the F-35 JPO may consider 
implementing in contracts, though they are not currently. 


Our Response 

The comments from JPO partially meet the intent of our recommendation; 
however, we disagree with JPO’s position that Lockheed Martin’s approach satisfies 
the F-35 program’s quality objectives as defined by the F-35 Program Quality 
Management Plan. According to the quality management plan, analysis of assembly 
defects and Quality Assurance Reports per aircraft is necessary to determine the 
progress in meeting the objectives. JPO needs to work with Lockheed Martin to 
identify quality metricfs) that could be used contractually to ensure that quality 
issues per aircraft are reduced to meet program goals. We request that JPO 
provide further comments in response to the final report. 
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2. Coordinate with DCMA to implement an effective root cause analysis and 
corrective action process in order to reduce assembly defect rate. 

F-35 Joint Program Office Comments 

F-35 Joint Program Office partially agreed and stated: 

There are effective Corrective Action processes already in place within 
Lockheed Martin and the supply chain. DCMA and F-35 JPO quality 
organizations focus heavily on continually improving these Systems and 
process to assure cause analysis leads to true root cause and does not stop 
with direct cause. This includes areas outside production, including work 
instructions, tooling, design, facilities, human resources and even within the 
government should government furnished equipment property or direction 
be a contributor. Lockheed Martin has also implemented internal corrective 
action processes for non-shop floor continual improvement opportunities. 
While F-35 JPO agrees effective root cause and corrective action are essential 
to improvement, beyond continuous high priority focus through Corrective 
Action Board meetings, metrics reviews, DCMA issuance of requests for 
corrective action no additional changes are necessary at this time. 


Our Response 

The comments from JPO do not meet the intent of our recommendation. Our intent 
was for JPO to take actions to identify and correct the cause of the program's 
inability to meet established quality goals and objectives for full-rate production as 
stated in the F-35 Program Quality Management Plan. JPO should work with DCMA 
and Lockheed Martin to reduce the number of assembly defects per aircraft though 
quality improvement initiatives. We request that JPO provide further comments in 
response to the final report. 
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Finding E 

Corrective Action Request Escalation Was Inadequate 

DCMA Fort Worth was not escalating unresolved CARs to the next higher level as 
required by the applicable DCMA instruction. If DCMA does not escalate CARs 
as required, deficiencies may continue to occur and DCMA and JPO leadership 
may not be aware of problems until they adversely impact the cost, Schedule, and 
performance of the program. 


Discussion 

DCMA was not following its policy to escalate CARs to the next higher level 
when repetitive Level II CARs have been issued, when CAPs were rejected 
and not resubmitted within the 10-day requirement, or when the corrective 
action validation is rejected. We identified instances when multiple CARs were 
documented for the same violation, but none were escalated to the next level. 

We also noted instances when CAPs were rejected and were not resubmitted 
within the 10-day requirement, but did not result in DCMA escalating the CAR to 
the next level. 

Management Comments on the Finding 
and Our Response 

Summaries of management comments on the finding and our response are in 
Appendix C. 
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Recommendations, Management Comments, 
and Our Response 

Recommendation E 

We recommend that the Defense Contract Management Agency: 

1. Review all unresolved CARs and escalate those that meet the criteria 
established in DCMA policies and instructions. 

DCMA Comments 

DCMA agreed and stated: 

DCMA agrees we did not execute to DCMA Instruction #1201 as interpreted 
by the DoD IG team. We agree that DCMA Instruction 1201 requires 
clarification. It is currently under revision to address clarification to the CAR 
elevation process. DCMA Headquarters will provide an updated policy with 
an estimated completion date of early 2015. Upon policy update approval, 
DCMA LMFW [Lockheed Martin, Fort Worth] will review all open CARS to 
determine if any warrant elevation to the next level as recommended. 


Our Response 

The comments from DCMA meet the intent of our recommendation. No further 
comments are required. 

2. Assess all CARs that were not properly elevated and assess any impact on 
the product. 

DCMA Comments 

DCMA agreed and stated: 

DCMA LMFW will assess all open CARs based on new policy guidance to 
identify, analyze, and assess any impacts to F-35 product. 


Our Response 

The comments from DCMA meet the intent of our recommendation. No further 
comments are required. 
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Finding F 

Software Quality Management Was Insufficiënt 

JPO and Lockheed Martin were not ensuring software quality management 
processes such as configuration management, failure reporting and corrective 
action, nonconformance root cause and analysis, and training were performed 
sufficiently to prevent delayed mission capability, software rework, and possibly 
additional program costs. In addition, undiscovered software defects could result 
in performance degradation or even catastrophic failure of aircraft, subsequently 
resulting in loss of aircraft and death. 


Discussion 

Lockheed Martin's software development methodology did not facilitate the 
performance of physical and functional configuration audits. The majority of 
new aircraft capability is based on additional software builds; therefore, it is 
necessary to perform functional and physical configuration audits. These types 
of audits ensure that CSCIs exist and are configured, tested, and documented in 
accordance with software specifications and requirements to mitigate aircraft 
performance and reliability risks. In addition, deficiencies in Lockheed Martin's 
software development training program may lead to knowledge gaps which could 
result in software development process failures and software defects. Additionally, 
the classification of the severity of software deficiencies identifies the level of 
verification and corrective action required. Lockheed Martin's failure to ensure 
software subcontractors, such as Northrop Grumman, were categorizing software 
deficiencies in accordance with the F-35 Air System Software Development Plan 
does not provide confidence in delivered software. Allowing open deficiencies, 
especially those whose severities are not characterized and understood, can result 
in performance degradation or catastrophic failure during operation. JPO needs to 
ensure that rigorous software processes exist at Lockheed Martin. 
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Management Comments on the Finding 
and Our Response 

Summaries of management comments on the finding and our response are in 
Appendix C. 

Recommendation, Management Comments, 
and Our Response 

Recommendation F 

We recommend that the F-35 Joint Program Office work with Lockheed Martin 
to ensure software quality management systems are improved; metrics 
should be reported on a periodic basis (for example, monthly) to evaluate 
process improvement. 

F-35 Joint Program Office Comments 

F-35 Joint Program Office agreed and stated: 

The F-35 JPO has been and will continue to work with Lockheed Martin 
to ensure the adequacy and improvement of software quality management 
processes and practices. Metrics reflecting software quality and quality 
improvement objectives have been developed and are being reported and 
assessed at both the vehicle systems and mission systems team levels on a 
monthly basis, as well as at the air vehicle level and production quality on a 
monthly basis. 


Our Response 

The comments from JPO partially meet the intent of our recommendation. JPO does 
not give the impression that any changes and/or improvement will be made to the 
software quality management process as a result of the finding. We request that 
JPO, in response to the final report, provide additional information regarding the 
types of improvements that have been implemented to show that critical software 
verification processes are implemented at various levels to ensure software 
product integrity. 
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Finding G 

Subcontract Management Procedures Were 
Still Inadequate 

JPO, DCMA, and Lockheed Martin were not ensuring that contract requirements 
were adequately flowed down, contract deliverables were evaluated, and minor 
nonconformances were approved by the proper authority. Lack of sufficiënt 
subcontract management activities may result in the acceptance of products that 
do not meet intended performance and reliability requirements. 


Discussion 

DCMA and Lockheed Martin were not ensuring that the purchase orders identified 
the specific revisions of the technical requirement documentation necessary for 
the production, testing, and inspection of each item being purchased and changes 
to technical requirements were contractually documented. They also did not 
ensure that contractually required supplier-generated data items were delivered by 
subcontractors when required. We found no evidence of any appropriate actions 
taken when subcontractors did not comply with these requirements. In addition, 
JPO and Lockheed Martin were not ensuring that disposition and approval of minor 
nonconformances were approved only by DCMA, who has the delegated authority 
for approval of minor nonconformances. 

Recommendations, Management Comments, 
and Our Response 

Recommendation G 

1. We recommend that the F-35 Joint Program Office ensure that all minor 
nonconformances are evaluated and approved only by DCMA. 

F-35 Joint Program Office Comments 

F-35 Joint Program Office partially agreed and stated: 

F-35 JPO will work with DCMA to assure compliance with regulation and 
direction. As part of an existing effort to validate requirements flow down 
accuracy minor nonconformance approval will be included. Scrap and 
rework-to-design-definition (so called rework to blue-print] dispositions 
ultimately result in conforming product being presented to the government 
for acceptance and alleviate the requirement of further DCMA review. DCMA 
has and will continue to intercede in repetitive scrap and rework. In the 
case of repair or use-as-is disposition, DCMA reviews these in accordance 
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with regulation and instruction as it results in nonconforming product being 
tendered for acceptance. The F-35 JPO will work with DCMA to assure 
nonconformance evaluations are properly reviewed in accordance with 
requirements and action plans put in place by March 31, 2015. 


Our Response 

The comments from JPO meet the intent of the recommendation. No further 
comments are required. 

2. We recommend that the Defense Contract Management Agency ensure 
that Lockheed Martin flows down the appropriate technical requirements 
to its subcontractors, and receives and evaluates contract deliverables 
within the required time frames. 

DCMA Comments 

DCMA agreed and stated: 

DCMA LM will assess current functional surveillance strategies to determine 
system/process surveillance gaps regarding Subcontract Management 
Procedures. DCMA LM surveillance strategies will be revised as necessary 
to ensure LM processes are adequate to provide appropriate flow down 
of requirements and execution of those requirements are achieved by 
their subcontractors. 


Our Response 

The comments from DCMA meet the intent of the recommendation. No further 
comments are required. 


DODIG-2015-092 | 39 



Appendixes 


Appendix A 

Scope and Methodology 

We conducted this evaluation from August 2014 through December 2014 in 
accordance with the Council of the Inspectors General on Integrity and Efficiency, 
"Quality Standards for Inspection and Evaluation." Those standards require that 
we plan and perform the evaluation to obtain sufficiënt, appropriate evidence to 
provide a reasonable basis for our findings and conclusions based on our evaluation 
objectives. We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our evaluation objectives. 

The scope of this evaluation was to determine the F-35 Program conformity 
to the contractually required AS9100, "Quality Management Systems - 
Requirements for Aviation, Space and Defense Organizations," Standard and 
whether the F-35 Joint Program Office (JPO), Defense Contract Management 
Agency (DCMA), and the prime contractor—Lockheed Martin Aeronautics 
Company took appropriate actions on findings and recommendations identified 
in the DoD IG Report No. DODIG-2013-140, "Quality Assurance Assessment of the 
F-35 Lightning II Program," September 30, 2013. We performed a two-week AS9100 
inspection at Lockheed Martin Aeronautics Company, Fort Worth, Texas, from 
September 22, 2014 to October 3, 2014. 

The AS9100C Standard breaks down quality assurance requirements into 
five major clauses: 

• Quality Management System, 

• Management Responsibility, 

• Resource Management, 

• Product Realization, and 

• Measurement, Analysis, and Improvement. 

The first three major AS9100 clauses, Quality Management System, Management 
Responsibility, and Resource Management, require the organization to have 
a quality assurance management organization that has all the resources and 
authority to affect the end-item quality of the product. In addition, it requires 
the organization to have a quality assurance manual and strict control over all 
documentation, data, and procedures that affect the quality of the product. 
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The fourth major AS9100 clause, Product Realization, covers the activities and 
processes necessary to bring a product into existence. It is broken down further by 
the AS9100 Standard as follows: 

• Planning of Product Realization, 

• Customer-Related Processes, 

• Design and Development, 

• Purchasing, 

• Production and Service Provision, and 

• Control of Monitoring and Measuring Equipment. 

Planning of Product Realization requires the organization to develop processes 
needed for design and development of product and includes elements such 
as procedures, quality assurance records, resource requirements, safety and 
reliability programs, and inspection and test. The Customer-Related Processes 
section requires the organization to determine all applicable requirements 
specified by the customer, understand those requirements and associated risks, 
and implement arrangements for communication with the customer. Design 
and Development includes requirements that cover planning, inputs, outputs, 
review, verification, validation, and control of changes as related to design and 
development. Purchasing requires the organization to ensure the purchased 
product conforms to specified purchase requirements and all products purchased 
from suppliers are verified against these requirements. Production and Service 
Provision requires the organization to ensure that production is accomplished 
under controlled conditions using drawings and specifications, work instructions, 
production tools and software programs, and provide evidence that all production 
and inspection/verification operations have been completed as planned. Control 
of monitoring and measuring equipment requires the organization to establish, 
implement, and maintain a process for managing, monitoring, and measuring 
equipment that requires calibration or verification. 

The last major AS9100 clause, Measurement, Analysis, and Improvement, requires 
the organization to ensure the product continuously improves. This clause includes 
customer satisfaction, internal audit, monitoring and measuring processes and 
product, and control of nonconforming products to ensure continual improvement. 
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Statutory and Regulatory Requirements 

Additionally, we evaluated the F-35 Program’s compliance with applicable 
statutory and regulatory requirements to include the Federal Acquisition 
Regulation; Defense Federal Acquisition Regulation Supplement; Interim DoD 
Instruction 5000.02/ "Operation of the Defense Acquisition System," enclosure 1; 
and DoD 4140.01, "DoD Supply Chain Materiel Management Procedures: Materiel 
Sourcing," volume 3. 

Aviation Chtical Safety Items Requirements 

Furthermore, we evaluated the F-35 Program’s implementation of aviation 
CSI requirements. Special attention should be paid to CSIs to prevent the potential 
catastrophic or critical consequences of failure. CSIs require special handling, 
engineering, manufacturing, and inspection documentation to control and ensure 
safety of flight. 

Public Law 108-136, Section 802, "Quality control in procurement of aviation 
CSIs and related services,” requires the DoD to prescribe a quality control 
policy for the procurement of aviation CSIs. The Joint Service CSI Instruction, 
"Management of Aviation Critical Safety Items," implements the DoD CSI Program 
and establishes the policies, procedures, and responsibilities to manage CSIs. 

The Joint Aeronautical Commanders Group (JACG), "JACG Aviation Critical Safety 
Item Handbook," supplements the Joint Service CSI Instruction and provides 
implementation guidance which describes the technical and quality assurance 
requirements for a prime contractor CSI Program. 

Use of Computer-Processed Data 

We did not use computer-processed data to perform this evaluation. 

Use of Technical Assistance 

We obtained assistance from quality assurance engineers and quality 
assurance specialists with a background in defense and aerospace Systems. 

We established teams of subject matter experts who evaluated the F-35 Program 
to the AS9100C, “Quality Management Systems - Requirements for Aviation, Space 
and Defense Organizations," Standard. The subject matter expert teams consisted 
of 17 quality assurance engineers, trained and certified in AS9100, who had an 
average of 17 years of quality assurance, audit experience. Additionally, our teams 
included a subject matter expert in military aviation CSI. 


7 At the time of our evaluation, DoDI 5000.02 was in interim status. The final DoDI 5000.02 was issued on January 7, 2015. 
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Appendix B 

Prior Coverage 

The Government Accountability Office (GAO) and the Department of Defense 
Inspector General (IG) have issued 32 reports discussing the F-35 Joint Strike Fighter. 
Unrestricted GAO reports can be accessed at http://www.gao.gov . Unrestricted 
DoD IG reports can be accessed at http://www.dodig.mil/pubs/index.cfm . 

GAO 

Report No. GAO-14-778, "F-35 Sustainment: Need for Affordable Strategy, Greater 
Attention to Risks, and Improved Cost Estimates,” September 23, 2014 

Report No. GAO-14-340SP, "Defense Acquisitions: Assessments of Selected Weapon 
Programs,” March 31, 2014 

Report No. GA0-14-468T, "F-35 Joint Strike Fighter: Slower Than Expected Progress 
in Software Testing May Limit Initial Warfighting Capabilities," March 26, 2014 

Report No. GAO-14-322, "F-35 Joint Strike Fighter: Problems Completing Software 
Testing May Hinder Delivery of Expected Warfighting Capabilities," March 24, 2014 

Report No. GAO-13-690T, "F-35 Joint Strike Fighter: Restructuring Has Improved the 
Program, but Affordability Challenges and Other Risks Remain," June 19, 2013 

Report No. GAO-13-500T, "F-35 Joint Strike Fighter: Program Has Improved in Some 
Areas, but Affordability Challenges and Other Risks Remain,” April 17, 2013 

Report No. GAO-13-294SP, "Defense Acquisitions: Assessments of Selected Weapon 
Programs,” March 28, 2013 

Report No. GAO-13-309, "F-35 Joint Strike Fighter: Current Outlook Is Improved, but 
Long-Term Affordability Is a Major Concern,” March 11, 2013 

Report No. GAO-12-437, "Joint Strike Fighter: DOD Actions Needed to Further 
Enhance Restructuring and Address Affordability Risks," June 14, 2012 

Report No. GAO-12-400SP, "Defense Acquisitions: Assessments of Selected Weapon 
Programs,” March 29, 2012 

Report No. GAO-12-525T, "Joint Strike Fighter: Restructuring Added Resources and 
Reduced Risk, but Concurrency Is Still a Major Concern," March 20, 2012 
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Report No. GAO-11-903R, "Joint Strike Fighter: Implications of Program 
Restructuring and Other Recent Developments on Key Aspects of DOD’s Prior 
Alternate Engine Analyses," September 14, 2011 

Report No. GAO-11-677T, "Joint Strike Fighter: Restructuring Places Program on 
Firmer Footing, but Progress Is Still Lagging,” May 19, 2011 

Report No. GAO-11-325, "Joint Strike Fighter: Restructuring Places Program on 
Firmer Footing, but Progress Still Lags," April 7, 2011 

Report No. GAO-11-450T, "Joint Strike Fighter: Restructuring Should Improve 
Outcomes, but Progress Is Still Lagging Overall,” March 15, 2011 

Report No. GAO-11-323R, "Tactical Aircraft: Air Force Fighter Force Structure 
Reports Generally Addressed Congressional Mandates, but Reflected Dated Plans 
and Guidance, and Limited Analyses," February 24, 2011 

Report No. GAO-11-171R, "Defense Management: DOD Needs to Monitor and Assess 
Corrective Actions Resulting from lts Corrosion Study of the F-35 Joint Strike 
Fighter,” December 16, 2010 

Report No. GAO-10-1020R, "Joint Strike Fighter: Assessment of DOD’s Funding 
Projection for the F136 Alternate Engine," September 15, 2010 

Report No. GAO-10-789, "Tactical Aircraft: DOD’s Ability to Meet Future 
Requirements is Uncertain, with Key Analyses Needed to Inform Upcoming 
Investment Decisions," July 29, 2010 

Report No. GAO-10-388SP, "Defense Acquisitions: Assessments of Selected Weapon 
Programs," March 30, 2010 

Report No. GAO-10-478T, "Joint Strike Fighter: Significant Challenges and Decisions 
Ahead,” March 24, 2010 

Report No. GAO-10-382, "Joint Strike Fighter: Additional Costs and Delays Risk Not 
Meeting Warfighter Requirements on Time,” March 19, 2010 

Report No. GAO-10-520T, "Joint Strike Fighter: Significant Challenges Remain as 
DOD Restructures Program,” March 11, 2010 

Report No. GAO-09-711T, "Joint Strike Fighter: Strong Risk Management Essential as 
Program Enters Most Challenging Phase," May 20, 2009 

Report No. GAO-09-326SP, "Defense Acquisitions: Assessments of Selected Weapon 
Programs," March 30, 2009 
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Report No. GAO-09-303, "Joint Strike Fighter: Accelerating Procurement 
before Completing Development Increases the Government’s Financial Risk," 

March 12, 2009 

Report No. GAO-08-782T, "Defense Acquisitions: Better Weapon Program Outcomes 
Require Discipline, Accountability, and Fundamental Changes in the Acquisition 
Environment," June 3, 2008 

Report No. GAO-08-467SP, "Defense Acquisitions: Assessments of Selected Weapon 
Programs," March 31, 2008 

Report No. GAO-08-569T, "Joint Strike Fighter: Impact of Recent Decisions on 
Program Risks,” March 11, 2008 

Report No. GAO-08-388, "Joint Strike Fighter: Recent Decisions by DOD Add to 
Program Risks,” March 11, 2008 

DoD IG 

Report No. DODIG-2013-140, "Quality Assurance Assessment of the F-35 Lightning II 
Program,” September 30, 2013 

Report No. DODIG-2013-031, "Audit of the F-35 Lightning II Autonomie Logistics 
Information System (ALIS)," December 10, 2012 
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Management Comments on the Findings 
and Our Responses 

Finding A - Insufficiënt Progress Had Been Made Toward 
Implementation of the Critical Safety Program 

F-35 Joint Program Office Comments 

F-35 Joint Program Office disagreed and stated: 

The F-35 JPO places the safety of aircrew and maintainers foremost within 
all program priorities. Working with DCMA and Lockheed Martin to bring 
the management of CSI into compliance with requirements, progress is being 
made toward the dates provided to the DODIG. This is not however, as 
implied, a simple matter of Tlowing down' this highly complex requirement. 
This contractual change constitutes an increase in scope, incorporating 
non-recurring and recurring effort into multiple F-35 contracts, and 
must be done in accordance with contract law, federal regulation and 
fiscal responsibility. 

Military aviation programs that existed before the CSI requirement was 
created have faced similar challenges implementing the CSI law based on wide 
interpretation of its intricate and far reaching mandate. The F-35 Lightning II 
System Design and Development Contract was signed October 26, 2001, by 
2003 the basic design was nearly complete and fabrication beginning, with 
first rollout in February 2006. Even recognizing aircraft design changes for 
weight reduction in 2004, when the CSI public law was issued it constituted 
a new requirement to the existing contract. Also, implementing such a 
requirement without department instruction or guidance could have incurred 
significant risk in implementing it incorrectly. The first issuance of the 
Joint Aeronautical Logistics Commanders (JALC) Aviation CSI handbook 
was August 12, 2005 and the Joint Service Instruction was not issued until 
January 25, 2006. 

As was objectively demonstrated in explicit detail to the DODIG the F-35 
JPO and Lockheed Martin jointly developed F-35 integrity programs based 
on several decades of military aircraft design and development for the 
United States armed forces. The integrity policies and procedures form one of 
the cornerstones for rigorous qualification of designs to ensure safety and 
functionality that progressively moves designs from engineering analysis 
(including flight safety critical aircraft parts) through production fabrication, 
testing, and lifecycle management before incorporation into the aircraft. 
Additionally, designs are secondarily analyzed by government subject matter 
experts for airworthiness and certified to standards developed specifically 
for this purpose. Next, the qualified parts are assembled to constitute an 
aircraft which moves into a series of reviews and certifications before first 
flight of a severely restricted flight envelope that is progressively opened 
through carefully monitored flight testing conducted by the contractor 
and government. 
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The CSI requirement establishes policy, procedure and assigns responsibility 
for the lifecycle management of replenishment items (spare parts, support 
equipment and repairs) critical to safety. The primary methodology is to 
assure spare parts and services are procured from approved sources and 
manufactured to correct standards and processes. The public law, department 
and service instructions do not specify that CSI is to be incorporated 
during the development phase of aircraft Systems, though it has been 
recognized within DoD that the intrinsically necessary failure mode analysis, 
documentation of characteristics, development of quality assurance above and 
beyond Standard practice with primary difference being removal of likelihood 
as analysis criteria should be done in parallel with other safety, airworthiness 
and integrity scrutiny. To incorporate CSI in this manner it would have to 
be done during a contract's initial proposal which in the case of F-35 was 
several years before the CSI law's inception. Therefore the F-35 JPO has 
used department instruction to interpret the program and existing drawings 
as legacy. The F-35 JPO engineering support activity considers there to 
be sufficiënt protections in place to assure safety through existing quality 
assurance, integrity, qualification, airworthiness, and flight test programs 
until full CSI compliance is reached. Currently all F-35 Lightning II spares 
and repairs are procured from their original equipment manufacturers with 
inherent quality, integrity, airworthiness and safety assured. 


Our Response 

JPO is planning to implement the CSI Program by February 2016; however, the 
draft CSI Program prepared by Lockheed Martin that we evaluated did not 
sufficiently address requirements described in the Joint Service CSI Instruction. 
The F-35 Program has not made sufficiënt progress toward full compliance 
with CSI Public Law since it was issued in 2003. NAVAIR Instruction 4200.25C, 
September 1999, had been established and in use prior to the SDD contract 
award in 2001. NAVIAR Instruction 4200.25D, which was issued in June 2002, 
contains very similar CSI requirements as SECNAVINST 4140.2, which was 
released in January 2006. There are many other DoD instructions and guidelines 
such as DoD Regulation 4140.1-R, May 2003 and the prior version, which was 
issued in January 1993; JALC CSI, August 2005; the Joint Service Instruction, 
January 2006; and JACG Aviation CSI Management Handbook, March 2011; 
which were available to be used for the F-35 CSI Program. These requirements 
could have been implemented in either the SDD or LRIP contracts, starting with 
LRIP lot 1. JPO was aware of the existence of NAVAIRINST 4200.25D as it was put 
in the LRIP lot 1 contract for F-35 engine development awarded in April 2006. 
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JPO believes that the F-35 integrity programs, which are jointly developed by 
Lockheed Martin and JPO, address all the CSI requirements; however, JPO was not 
able to provide any evidence to show that it had performed a gap analysis between 
the F-35 integrity programs and CSI requirements. We identified gaps during our 
last evaluation, which showed that the F-35 integrity program did not cover all 
CSI requirements. 

JPO stated,"[t]he public law, department and service instructions do not 
specify that CSI is to be incorporated during the development phase of aircraft 
systems...” Contrary to JPO’s claim, JACG Aviation CSI Management Handbook, 
section 2.3.2 States: 

[tjhe basis of CSI determinations on repairable and consumable 
parts and maintenance concepts underscores the importance of 
effective working relationships among logistics and engineering 
communities. Logisticians are responsible for life cycle support 
concepts that ultimately define spare parts and for providing these 
concepts to engineers who are then responsible for identifying 
CSIs and approving their sources. This relationship is particularly 
meaningful during the acquisition, development, and initial 
production stages of new platforms. 

Section 2.9.3 of JACG Aviation CSI Management Handbook further States, 
"[tjherefore, a requirement for prime contractor/OEM's to identify critical 
characteristics for every new CSI should be included in the System 
Development and Demonstration Statement of Work for new systems 
and major platform modifications.” 

JPO’s decision to postpone the implementation of CSI requirements may increase 
cost, Schedule, and safety risks. Significantly more efforts and resources will 
be required to accommodate CSI requirements if they are implemented in the 
later stages of the acquisition lifecycle: such as performing engineering analysis, 
identifying critical characteristics, updating drawings, and creating a logistics 
process to support the changes. JPO did not explain how it is more fiscally 
responsible to delay incorporating contractual CSI requirements. In addition, 

JPO had many opportunities to implement CSI requirements during program 
restructuring, which took place in December 2003, again in March 2007, and most 
recently in March 2012. 
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Finding B - Oversight of Product Development and Realization 
of Requirements was Inadequate 

F-35 Joint Program Office Comments 

F-35 Joint Program Office disagreed and stated: 

The F-35 JPO has progressively made improvements to aircraft acceptance 
criteria. In LRIP 6, as in previously LRIPs, known JCS non-compliances 
were documented. It is true that at aircraft acceptance F-35 JPO does 
not require LM to positively verify compliance to every JCS requirement. 

This is due to the concurrent nature of SDD and Production schedules. 

The SDD schedules identify when each JCS requirement is expected to meet 
compliance criteria. The vast majority of compliance occurs in the 2016 and 
2017 timeframe. If F-35 JPO were to impose JCS compliance assessments in 
production, prior to expected SDD JCS compliance dates, the assessments 
would be based on subjective rather than objective criteria because proper 
SDD certification evidence would not be available. The F-35 JPO deemed that 
subjective, premature JCS compliance assessments were not value add for the 
program. Additionally, these subjective assessments would negatively impact 
SDD resources (people, labs and flight test assets) and schedules. 

Our Response 

Acquisition strategy (concurrent development) should not prevent JPO from 
establishing objective and measurable aircraft acceptance criteria. The lack of 
objective and measurable aircraft acceptance criteria in the CCDD was documented 
during our last evaluation. This nonconformity was agreed to by Lockheed Martin 
and JPO; however, JPO still has not addressed it adequately. This same deficiency 
was documented again during this evaluation. Our report States “...JPO and 
Lockheed Martin are at an impasse to perform implementation of corrective 
action due to a breakdown in contract negotiations." As a result, there has not 
been sufficiënt progress in defining objective and measureable aircraft acceptance 
criteria. Our finding does not state that the F-35 LRIP aircraft need to be 
tested at full JCS compliance. Rather, the CCDD acceptance criteria for a specific 
LRIP lot should be measurable and testable. We understand that production 
and development efforts are concurrent; however, it does not relieve JPO of the 
responsibility for identifying clear and measureable acceptance criteria for the 
LRIP aircraft. 
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Finding C - The Quality Assurance Organization Was Not 
Independent and Not Adequately Staffed 

F-35 Joint Program Office Comments 

F-35 Joint Program Office partially agreed and stated: 

The F-35 JPO does not place schedules ahead of quality. Further, the F-35 JPO 
has withheld delivery of aircraft to assure quality issues are addressed. 
Although the F-35 JPO deerns the current reporting structure adequate to meet 
program needs, it has agreed to create a more robust quality organization to 
ensure quality management is objectively measured and reported. 


Our Response 

The F-35 Program quality assurance organization still resides under the Director 
of Production and is not a direct report to the F-35 PEO. We recommend that 
JPO work toward the organizational structure defined in JPO’s response letter 
to the DODIG-2013-140 report dated August 8, 2014. The response letter stated, 
"[t]he F-35 JPO will establish an independent quality organization reporting to the 
Program Executive Officer." However, during the 2014 evaluation, JPO’s planned 
quality organization reported to the Director of Production, which we do not 
consider to be independent. JPO should realign its quality organization in 
accordance with its August 8, 2014, response. 

Finding D - Reduction of the Assembly Defect Rate 
Was Inadequate 

F-35 Joint Program Office Comments 

F-35 Joint Program Office disagreed and stated: 

The program quality 90% objective the DODIG references was a ratio of Scrap 
Rework and Repair (SRR) to production labor hours (hence a 'rate'). This 
is a normalization technique to assess levels of SRR relative to volume of 
work in a given area. This normalization is a valuable metric; however, it is 
not the only metric. In the current case of F-35 the production line which is 
still experiencing significant production labor hour reductions as a result of 
progressing down predictable learning, the ratio of declining SRR hours to 
declining production labor hours has remained relatively steady. The F-35 JPO, 
DCMA and LM have worked to understand this phenomenon and analyzed 
other metrics to evaluate the program's progress toward reductions necessary 
to reach high production rates by increasing efficiency and flow. 

• Factory SRR hours have decreased 65% since the first production lot. 
Supplier responsible SRRs have been reduced 80% and the program is 
setting a goal to reduce the current level by another 25% again this year. 

• Quality Escapes to the field are another measure, which have decreased 
75% since 2011. 
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Our Response 

We appreciate the additional clarification on the full rate production objective 
identified in the F-35 Program Quality Management Plan. However, the Quality 
Management Plan does not provide any reference to the use of SRR to determine 
the assembly defect rate. Rather, it depicted a reduction in the number of quality 
assurance reports per aircraft per LRIP lot as a reduction in the defect rate, which 
is not equivalent. 

Finding E - Corrective Action Request Escalation 
Was Inadequate 

DCMA Comments 

DCMA agreed and stated: 

DCMA agrees we did not execute to DCMA Instruction #1201 as interpreted 
by the DoD IG team. We agree that DCMA instruction 1201 requires 
clarification. It is currently under revision to address clarification to the 
CAR elevation process. 

DCMA believes this risk is mitigated based on a comprehensive program 
quality strategy between DCMA, F-35 JPO, and LM. There is a monthly 
metrics cadence in which the DCMA Quality organization provides 
independent CAR analysis. The data presented in these meetings provides 
insight of non-compliances identified through DCMA and LM Quality 
surveillance activities. 


Our Response 

Comments from DCMA meet the intent of the finding. 

Finding F - Software Quality Management Was Insufficiënt 

F-35 Joint Program Office Comments 

F-35 Joint Program Office disagreed and stated: 

The F-35 JPO considers F-35 software quality management is sufficiënt, 
specifically the air vehicle software quality referenced in the draft DOD IG 
audit report. The F-35 JPO is confident current air vehicle software 
configuration management, failure reporting and corrective action, root cause 
analysis, and training are sufficiënt and adequate to support program mission 
capability and cost objectives. The F-35 JPO is confident in the current 
air vehicle software, systems engineering, and airworthiness certification 
processes will ensure no undiscovered software defects will be delivered 
with the air vehicle that could result in performance degradation or even 
catastrophic failure of aircraft. 
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Specific DoD IG Finding Responses: 

1. DoD IG Finding: Lockheed Martin's software development methodology 
did not facilitate the performance of physical and functional 
configuration audits. The majority of new aircraft capability is based 
on additional software builds; therefore, it is necessary to perform 
functional and physical configuration audits. These types of audits 
ensure that CSCIs exist and are configured, tested, and documented in 
accordance with software specifications and requirements to mitigate 
aircraft performance and reliability risks. 

F-35 JPO Response: In accordance with the F-35 program Software 
Development Plan and Configuration Management Plan, physical 
and functional configuration audits of the type referenced by the 
DOD IG report are conducted against the air vehicle Software 
Data Load, which is a compilation of vehicle and mission systems 
components. The program is conducting FCA/PCAs in accordance 
with the F-35 CM Plan, 2YZA00002, on all Cis and CSCIs identified in 
the F-35 Cl List, 2YZA00941, approved by the F-35 JPO. This finding 
erroneously presumes that certain software items which are not 
identified in the Cl list are CSCIs and therefore must be subject to 
FCA/PCA requirements. Mission System and Vehicle System computer 
programs are elements which contribute to the composition of the 
aircraft Software Data Load. It is the aircraft Software Data Load 
which is identified in the F-35 Cl List as a CSCI, and it is this software 
product on which FCA/PCA activity will be conducted. This practice 
is consistent with software development practices on all weapon 
systems developed at the Fort Worth facility, including the F-16 and 
F-22, in accordance with the DoD required Single Process Initiative 
for the Fort Worth Facility. The F-35 program has a very rigorous and 
structured verification and validation process for software enabled 
systems capabilities, including the identification and management of 
all software components that contribute to safety critical functions. 

In addition, F-35 air vehicle software undergoes extensive independent 
reviews by external airworthiness agencies from the United States 
Air Force and Navy in the context of MIL-HDBK-516, as well as the 
Ministries of Defense of the United Kingdom and Australia. These 
agencies have independently confirmed the adequacy of the physical 
configuration and functional ability of the air vehicle software for safe 
operation in the delivered operational envelope. The F-35 program 
does not intend to modify existing functional and physical configuration 
audit practices for F-35 air vehicle software based on this finding. 
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2. DoD IG Finding: Deficiencies in Lockheed Martin's software 
development training program may lead to knowledge gaps which could 
result in software development process failures and software defects. 

F-35 JPO Response: Lockheed Martin has a structured and accountable 
training program for all software engineers in the LM aeronautical 
division, documented in PM-4002 (Software Engineering Training). 

In addition, the LM F-35 program has a structured and accountable 
training program for F-35 specific software development environments 
and tools. Training history for both LM Aero and F-35 specific training 
were provided to the IG team, as well as learning plans for specific 
classes of software engineering disciplines. Air vehicle software 
engineers on the F-35 program comprise a body of world class 
talent - they are industry experts in the development of hard real 
time embedded flight and mission critical software, and have in most 
cases decades of experience in vehicle and mission systems software 
development on other 4 th and 5 th generation fighter programs. They 
are sought after by industry to provide instruction to other engineers 
in their discipline areas, and work directly with the developers of 
the software tools and environments they are using to incorporate 
additional enhancements and features to increase productivity and 
efficiency. The F-35 JPO does not intend to pursue any additional 
training initiatives for the LM software development team based on 
this finding. 

3. DoD IG Finding: The classification of the severity of software 
deficiencies identifies the level of verification and corrective action 
required. Lockheed Martin's failure to ensure software subcontractors, 
such as Northrop Grumman, were categorizing software deficiencies 

in accordance with the F-35 Air System Software Development Plan 
does not provide confidence in delivered software. Allowing open 
deficiencies, especially those whose severities are not characterized 
and understood, can result in performance degradation or catastrophic 
failure during operation. 

F-35 JPO Response: System and software deficiency severity 
classification and tracking is managed and maintained at the air vehicle 
system level, in accordance with F-35 program systems engineering, 
software engineering, and system verification plans. The DoD IG 
team assumed subcontractors would independently classify and track 
their anomalies using the same criteria as the air vehicle teams, which 
is not the defined process or practice. Subcontractor deficiencies 
are reported against the specification requirements levied to the 
subcontractors, but cannot be categorized by the subcontractor team in 
terms of air vehicle severity, as only the air vehicle systems engineering 
teams in the vehicle systems and mission systems functional areas 
have the required breadth and scope of system functionality and safety 
criticality to adequately classify the severity of component anomalies 
in the context of air vehicle operations. The F-35 JPO reviewed the 
specific concerns highlighted by the DoD IG with Northrop Grumman 
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failure reporting, and found the reporting practice consistent with the 
documented air vehicle software, systems engineering, and air vehicle 
verification plan. There are no open deficiencies delivered with the air 
vehicle that are not characterized by hazard severity and understood 
within the context of mission and safety criticality, and which have 
not been adjudicated with the F-35 JPO, airworthiness authorities, and 
operational users. The F-35 JPO does not plan to modify the existing 
software, systems engineering, and system verification process or 
practices based on this finding. 


Our Response 

We do not agree that the F-35 Program’s software quality management program 
sufficiently meets the requirements of AS9100 and that no further corrective action 
is required. 

At the Air Vehicle Software Data Load (SDL) level, the FCA/PCA does not include all 
prime mission software configuration items such as: Fusion Software, Pilot System 
Software, Fire Controls/Navigation/Stores (MSFCS), Mission Domain (MSMIS), 

Data Collection Software, External Communication Software (MSXCM), and Core 
Processing Software. Eliminating the FCA/PCA program at the software level 
prevents the critical verification process and significantly impacts software 
product integrity. The intent of FCA/PCA cannot possibly be met at the Air Vehicle 
SDL level. 

Although Lockheed Martin’s training program did include position specific 
training requirements, we did not find any evidence that Lockheed Martin’s 
software integrated product teams identified and included project-specific training 
requirements within software engineer training curriculums. We also did not find 
any evidence that Lockheed Martin Integrated Product Team leads had reviewed 
training curricula annually as required by Lockheed Martin’s command media. 

Finally, we agree with JPO’s statement that F-35 Air System software deficiency 
severity levels may not apply directly to software defects within lower level 
software components. However, the Air System SDP stated "[p]roduct-level SDPs 
must be created or updated to conform to processes identified in this version of 
the AS [Air System] SDP." This identifies that the subcontractors SDP must comply 
with Air System SDP processes, which would include the categorization of software 
deficiencies to alleviate translation conflicts of the F-35 Air System software 
deficiency severity levels. 
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Management Comments 

F-35 Joint Program Office and Defense Contract 
Management Agency Comments 


F-35 LÏGHTNING II JOINT PROGRAM OFFICE 
200 I2 ,b Street South, Suite 600 
A rling ton, Virginia 22202-5402 

JAN 20 2015 

From: Program ExecuLive Ofïïcer, F-35 Lightning [J Joint Program Office (JPO) and Director, 
Defense Contract Management Agency (DCMA) 

To; Department of Defense (DoD) Deputy Inspector General (1G) ibr Polïcy and ü verst ght 

Subj: RESPONSE TO F-35 LIGHTNING tl PROGRAM QUAUTY ASSURANCE AND 
CÖRRECTTVE ACTION EVALUATiON (Project No. D2G14-OTOTAD-0004) 

Thank you for the opportunity to comment on your dm ft report, ^Quality Assurance 
Assessment of the F-35 Lïghtmng lï Program"* dated 23 December 2014. 

Both the F-35 Lightning IF Joint Strike Fighter Program Office (JPO) and ihc Defensc 
Contract Management Agency (DCMA) tafce very seriously the findings and recommendulions 
offered in the report, As noted in Enclosure l, F-35 JPO and DCMA Response to Findings and 
Recümmendations, our team bas aggressively addressed DoD FG fmdings since receipt of the 
first report in September 2013, The implementation of these corrective actions has led to 
significant progress in ddivering a quaiily, timely, and cosï-effective weapon system to the 
United States and its allics. 




Encïosure F provides a demi led response lo each tmding and recommendation. 

Enclosure 2 is a copy of a Lockheed Martin letter provided to the F-35 JPO detailing the quality 
improvement made sè nee issuance of the September 2Ö13 report. 

_Th:: r.lik-U-S Mi !iV-. 1 | HS■_ .IR- A'&th'd 



Lïcutenant General, USAF 
Program Executive Officer 


' 1 xü-VeJ 

WENDY M. MASIELLO 
Lieulenant General, USAF 
Defense Contract Management Agency 


Endosures: 

3, F-35 IPO and DCMA Response to Fmdmgs/Recommcndaüons 
2. LM Letter to F-35 JPO on Qua! ity Fmprovemenis 


uc: 

Office of the Under Secreiary of Defense for Acquisitton, Technology, and Logtstics 
Assistadt Secrdary ofthe Navy for Research, Development and Aequisition 
Assistent Seerctary ofthe Air Force (Acquismon) 
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F-35 Joint Program Office and Defense 
Contract Management Agency Response to 
Findings/Recommendations 


F-35 Ljplilfffntfir iind Qirrrrtive* Adkm lUfittttAi 

Fn4ffd Nn* 02014 OT0TA0*0004,000 
f-T5 \V0 mul ULMA atid ftusponscs 

Finding A 

Insufficiënt Progress Had Been Made Toward 
Implementation of the Critical Safety Item Program 

The F-35 Program had nol made suffictenL prqgress towards full compliance wEth Public Law it)8- 
136, Section 802, "Quality control in procurement of aviatron CSls and re la led services,'' and the 
Joint 5ervice CSI [nstmctkin, 'Management of Aviation Critical Safety Items/' by February 2016. 

This was catised by the lack of CSI requirements on the F-35 contracts, as required by public law 
and the Joint Service CSI ïnstruction. This lack of compliance with CSI public law and implementing 
instructions may impose an incrcased safety of flight risk on delivered aircraft 

Discussion 

As identified in our previous report JPO had not flmved down a contractual requirement to 
Lockheed Martin Lo implementa CSI Program in accordance with Public Law 108- 136, Section 802 
and the joint Service CSÏ Instruction. The CSI public law has been imposed since ZG03. 

JPÖand Lockheed Marün have taken iniLial steps to establisha CSI Program; however, Lhe program 
was suil not fully compüant with applicahie laws and rcgulaüons. JPO was planning to contractually 
require implemenLaLion of a CS1 Program by February 2016. 

Ho wever, the current draft of the CSÏ Program prepared hy Lockheed Martin did not sulficiently 
address requirements described in the Joint Service CS1 Instmction including the need to define 
criticaI characteristics for all CSJs and the proper limitations and deJegations for the materiai 
review board auihority far nonconforming CSI critical characteristics. 

F-35 JPO Response: Disagree 

The F-35 JPO places the safety of aircrew and maintainers foremost within all program 
prïonties. Working with DCMA and Lockheed Martin to bring the management of CSI 
into compliance wïth requirements, progress is being made toward the dates provided to 
the DODIG. This is not however, as implied, a simpfe matter of Tlowing down 1 this 
htghly complex requirement This contractual change constitutes an increase in scope, 
incorporating non-recurhng and recurring effort into multiple F-35 contracts, and must 
be done in accordance with contract law, federal regulation and fiscal responsibility. 

Military aviation programs that existed before the CSI requirement was created have 
faced simiiar challenges implementing the CSI law based on wide interpretation of ïts 
intricate and far reaching mandate. The F-35 Lightning II System Design and 
Development Contract was signed October 26, 2001, by 2003 the basic design was 
neariy complete and fabrication beginning, with first rollout in February 2006. Even 
recognizlng aircraft design changes for weight reduction in 2004, when the CS1 public 
law was Èssued rt constituted a new requirement to the existing contract. Also, 
implementing such a requirement without department instruction or guidance could 
have incurred significant risk in implementing it incorrectly. The first issuance of the 
Joint Aeronautical Logistics Commanders (JALC) Aviation CSI handbook was August 
12, 2005 and the Joint Service Instruction was not issued until January 25, 2006 

DISTRI BUTIGIM STATEMENT A. Approved for public release; distribution is unlimïted 
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F-35 Joint Program Office and Defense 
Contract Management Agency Response to 
Findings/Recommendations (cont'd) 


‘T I i i ni nuïï Program .11 'hi r t rcUVft Act 4 t i . i ■ ■ ■ 

Prolicl Wpi O2QM*nTQTAD*fl<l04.ua<l 
F-U5 )!"0 »mil UCMAl ; )mljr^JtL*eO(rm)tn*hi0^ni Ki^p*in9C& 

As was objectively demonstrafed in explicit detail to the DODIG the F-35 JPO and 
Lockheed Martin jointly devefoped F-35 integrity programs based on several decades of 
military aireraft design and developmem for the United States armed forces The 
integrity policies and procedures fonm one of the cornerstones for rigorous qualification 
of designs to ensure safety and fundionality that progressively moves designs from 
engineering analysis (induding flight safety critical aïrcreft parts) through production 
fabrication, testing, and lifecycle management before incorporation into the aircraft 
Additionally, designs are secondarïly analyzed by govemment subject matter experts for 
airworthiness and certified to standards developed speciftcally for this purpose. Next f 
the qualified parts are assembled to constitute an aircraft which moves info a series of 
reviews and certifications before first ffight of a severely restricted flight envelope that is 
progressively opened through carefully monitored flight testing conducted by the 
contractor and government 

The CSI requirement establishes policy, procedure and assigns responsibility for the 
lifecycle management of replenishment items (spare parts, support equipment and 
repairs) critical to safety. The primary methodology is to ass ure spare parts and 
services are procured from approved sources and manufactured to correct standards 
and processes The public law, department and service instructions do not specify that 
CSI is to be incorporated during the devefopment phase of aircraft Systems, though it 
has been recognized within DoD that the intrinsically necessary failure mode analysis, 
documentation of characteristics, development of quality assurance above and beyond 
Standard practice with primary difference being removal of likelihood as analysis criteria 
should be done in parallel with other safety, airworthiness and integrity scmtiny To 
incorporate CSI in this manner it would have to be done during a contract's initial 
proposal which in the case of F-35 was several years before the CS1 law's inception 
Therefore the F-35 JPO has used department instruction to interpret the program and 
existing drawings as legacy. The F-35 JPO engineering support activity considers there 
to be sufficiënt protections in place to assure safety through existing quality assurance, 
integnty, qualrïication, airworthiness, and ffight test programs until full CSt comptiance is 
reached Currently all F-35 Lightning tl spares and repairs are procured from their 
original equipment manufacturers with inherent quality, integrity, airworthiness and 
safety assured. 


Recommendations 

A. We recommend that the F-35 |oinL Program Office: 

L Ensure that the F-35 CSJ Program Is compliani with Public Uw 108 136, Section 802, "Qualïty 
controï Ut proeurement of aviation CS Is and related services/' and the joint Service CS1 Jnstruction, 
'Management of Aviation Critical Safety Items/ 1 

F-35 JPO Responsei Agree 


DlSTRiBUTlOIN STATEMENT A. Approved for public release; distri bution is ünlimited 
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F-35 Joint Program Office and Defense 
Contract Management Agency Response to 
Findings/Recommendations (cont'd) 


F-35 Lightmng II Program Quality Assurance and Corrective Action Evaluation 
Project No. D2Q14-DTOTAD-0004.000 
F-35 JPÜ and DCMA Findings, Recommendations and Responses 

- Contract actions for both SDD and LRIP that address the Joint Service CSI 
Instruction, “Management of Aviation CSIs” are in work or in place to implement the 
full F-35 CSI Program 

o SDD RFP for CSI NRE efforts to initiate F-35 CSI Program released Oct 2013 and 
resulted in a LM delivered Draft CSI Program Plan coordinated with F-35 JPO for 
apprcval of the CSI implementation approach and go ahead for delivery of the SDD 
proposal expected; go ahead date for proposal development expected 2QFY15 

- After the SDD CSI NRE efforts are completed, the full scope of the F-35 CSI 
program will be included in future production and su sta in ment contracts. 

o LRIP Production contracts (LRIP 1-8) currently cover explicït CSI tasks for "limited" 
compliance for several CSI ‘gaps' from the Joint Service CSI Instruction, 
Management of Aviation CSIs." 

- Maintain / update the DOE approved CSI list for configuration control 

- Use existing recurring processes: Materia! Review (MR), Variance, Item 
Acceptance, New Suppliers, Process Metrics, and Minor Si Major Change 
processes for CSIs 

- Provide CSI notification to affected suppliers via contracts letter 

- Provide notification if any identified non-conformances results in a safety impact 

- Report MR actions & QARs associated with CSI once per month 

- Use existing disposal processes to destroy scrapped CSIs that render part 
nonfunctional 

- DCMA-LM FW Integrafed CSI Surveillance Strategy developed Jan 2014 and covers 
CSI Delegations to ensure appropriate DCMA process surveillance of CSI parts, and 
imposed Government Source Inspection (GSI) on all parts that have been identified as 
CSIs per the Joint Service CSI Instructïon 


2.Conduct pcriodic CS1 Program evaluations of Lockheed Martin and Lheir suppliers Lo ensure 
compliance with public law and the Joint Service CSI fnsLrucüon. 

F-35 JPO Response: Agree 

F-35 JPO & LM conducted a CSI TIM in Nov 2014 following completion of the DoD IG 
Quality Assurance & Corrective Action Evaluation to address DoD IG “Opportunity For 
Improvemenf comments issued for CSIs and CSI Program Plan. 

F-35 JPO n DCMA & LM are conducting bi-weekly telecoms and periodic TIMs on-site at 
LM until SDD contract agreement is reached to implement the full scope CSI Program 
i n fra stru ctu re a n d p rocesse s 

After F-35 full scope CSI implementation is underway, the F-35 JPO wil! transition to a 
monitoring phase with periodic reviews, which will leverage or combine with other 
JPO/DCMA activities (joint audits, Production Readiness Reviews, quality reviews, 
metrics, etcetera). 
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Finding B 

Oversight of Product Development and Realization of 
Requirements Was Inadequate 

ƒ PO was noladequately managïngthe F-35 Program Lo ensure LhaLal! sysLemdevel requiremenU 
and capabilïdes delmed in the [SF JCS were realized and verifled. As a resutt, delivered and ïuture 
aircraft may not satisfy all system capability requirements and it is unclear to what extent these 
capabilities wil] be achieved by the compietion of the SDD phase, 

Discussion 

TheJCS document identiHes more Lhan 400 systemdeveJ requirements thatdefme performance 
characteristics. JPO and Lockheed Martin jointly IdentiHed 21 system JeveJ requirements contained 
wiLhin the JCS that were unltkeiy to be achieved by the end of the SDP phase of Lhe program, 
However, JPO and Lockheed Martin did not consider these 21 rcquirements as risks and did not 
handle them in accordance with tlieir risk management process. 

In addition, }PO still has not defined objective and measureable aircraft acceptante criteria for each 
LRIP lot The confirmation criteria defined in the CCDD did not indüde clearly measurable and 
verifïable acceptante criteria Lhat were ciearly traceable back U> the JCS* Th is was documented in 
our previous evaluatinn and contmues Lo be a challenge. 

F-35 JPO Response: Disagree 

The F-35 JPO has progressively made improvements to aircraft acceptance criteria, In 
LRIP 6, as in previously LRIPs r known JCS non-compliances were documented lt is 
true that at aircraft acceptance F-35 JPO does not require LM to positively verify 
compliance to every JCS requirement. This is due to the concurrent nature of SDD and 
Production schedules The SDD schedules identrfy when each JCS requirement is 
expecfed to meet compliance criteria. The vast majority of compliance occurs in the 
2016 and 2017 timeframe. If F-35 JPO were to impose JCS compHance assessments 
in producüon, prior to expected SDD JCS compliance dates, the assessments would be 
based on subjective rather than objective criteria because proper SDD certificatïon 
evidence would not be available. The F-35 JPO deemed that subjective, premature 
JCS compliance assessments were not value add for the program. Additionally, these 
subjective assessments would negatively impact SDD resources (people, labs and flight 
test assets) and schedutes 


Recommendations 

R We recommend tb at the F-35 Joint Program Office; 

1. Clearly dcllne coniractua! criteria fnr Lhc acceptante of all future and llcided aircroft Lo ensure 
that aircraft capabilities are verified. 
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F-35 JPO Response: Agree 

F-35 JPO is progressively refining acceptance criteria with each LRIP lot. The use of 
CCDDs will continue to be used for LRIP 7 and LRIP 8. These CCDD items require 
positive verification activity to occur, often with dedicated flight test evaluations; and 
they provide a means to ensure the weapons System meets warfighter expectation for 
that production lot. However, the use of these documents does not negate the 
responsibility to document any known JCS non-compliances at the time of aircraft 
acceptance. This will continue to occur as it has occurred in the past. As SDD 
progresses F-35 JPO does expect more qualification evidence to be available which will 
allowfor more robust docu mental ion of JCS non-compliance at the time of aircraft 
acceptance. This will be done in addition to requiring positive verificatïon for all CCDD 
items. 

Beginning with LRIP 9, the next lot to be negotiated, F-35 JPO will require full JCS 
compliance. F-35 JPO will assess LRIP 9+ product acceptance against the SDD 
specification. For future full-rate production contracts, there is planned a separate 
production specification that will leverage the final SDD specification 

2. Ensure that all 21 system-levcl requirements, in addidon to the risks associated with these 
capabilities, are documented, tracked, and mitigated usingthe established risk management 
process. 

F-35 JPO Response: Partially Agree 

F-35 JPO sees merit in tracking to closure the contractually defined 21 system level 
requirements that may not be fully met. F-35 JPO agrees the high-level visibility of a 
formal tracking Action Item to pertodically status these requirements to closure will 
ensure resolution of them in an expeditious manner. 

However, F-35 JPO DISAGREES with the recommendation to track these 21 system 
level requirements through the risk management process. These 21 requirements are 
not risks - they are a subset of requirements where the program has jointly 
acknowledged that the level of specified performance will not be met. Conversely, the 
purpose of risk management is to highlight and take proactive actions to avert a 
potentially bad outcome, such as a specification miss. Within the F-35 risk 
management process, risks are managed via a series of funded mitigation steps and 
technical gates to eliminate or reduce the likelihood and/or consequence of each risk. 
With these 21 requirements, F-35 JPO has already acknowledged the specification 
misses and has a contractual agreement to spend no more money to improve the 
design - using the cost as an independent variable principle for the requirements. The 
only remaining program resources associated with these 21 requirements is for 
gathering qualification evidence to verify the capability of the design as is. When 
qualifications are complete, F-35 JPO will resetthe specification requirements to the 
achieved performance and negotiate consideration for each of the shortfalls. 
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A closure tracking / status Act ion item for these 21 system Ie vel requirements wou ld 
regularly assess progress toward reconciliatlon and dosure of the spedfication to 
realized performance: qualification actions, coordination of venfication evidence, draft 
specificatiën change request engineering review. Configuration Steering Board 
approval, Change Control Board approval and contract negotiations / contract 
modifications. However, for these 21 system level requirements, nelther risk mitigations 
nor design changes are funded within the remaining scope of the System Design and 
Development contract As an aside, several of the 21 system level requirements are 
separately tracked as Program Technical Performance Measures 


Finding C 

The Quality Assurance Organization Was Not 
Independent and Not Adequately Staffed 

JPÜ had not defined its quality assurance organization's mies and responslbilities and had not 
provided adequate staffing to pevform effective oversight of the F-35 Program. 

Also f the quality assurance organization reports to the Director of Production rather than directly 
Lo the F‘35 PFÜ. The |PO qua!ity assurance fu netton did nat have the necessary independence fo 
implement the quality management mission fordevelopment produclïon,and susta inment ol the 
system. A quality assurance organizalion thallacks independente orthe inhcrenL authority to 
enforce quality management requirements may not effectively mitigate cost, Schedule, and 
performance risks to the F-35 Program. 

Discussion 

Qur previoiis evaluation recnmmended Lhat |P0 estabtish an independent quality assurance 
organization. ]PQ iniUated the establishment of a quality organization, assigned a Quality Lead. and 
plans to further develop and staff the orgamzation, But, the organization reports to the Director of 
Production rather than directly to the F-35 PEO, Because the desire to meet production sthethdes 
can often prevent adequate or sufficiënt root cause analysis and correctivo action. a quality 
assurance organizaUon that is independent and reporls to the PEO ensures greater influence in 
program dedsions. The current organizational structure may limit the quality assurance 
organizalion's ability to objectively report program qualiLy management performance. 

F-35 JPO Response: Partially Agree 

The F-35 JPO does not place schedules ahead of quality. Further, the F-35 JPO has 
withheld delivery of aircraft to assure quafity issues are addressed. Although the F-35 
JPO deerns the current reportïng structure adequate to meet program needs, it has 
agreed to create a more robusf quaüty organization to ensure quality management is 
objectively measured and reported. 

Recommendation 
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C. Wc recommend Lhat the F-35 Joint Program Office realign Lhc qualily assurancc organr/alion La 
report di reet ly to the PEO f dofine the organjznttón's roles and responsibilltles, and staff the 
organization apprapriately. 

F-35 JPO Response: Agree 

Efforts to establish an independent quality organization are on track As documented in 
F-35 JPtTs August 2014 response and reiterated during presentations and discussions 
with DODIG team members, A quality organization will be ïn place 12-24 months from 
the May 2014 decision. 


Finding D 

Reduction of tbc Assembly Defect Rate Was 
inadequate 

JPO and DCMA management werc nat committed to ensuring that assembly defect rate objectives 
and goals were realized, Lack of ability to reduce defects and the scrap, rework, and repair rate in 
accordance with defined program objectives will negatively impact program cost and Schedule, 

Discussion 

IPO was not ensuring thai Lockheed Martin was fuldlbng Lts commitment to a 90-percent assembly 
defect rate reduetton per aircraft to meet the fulhrate productkm goal of completing one aircraft 
per day, We were not able to determine Lockheed Martin J s progress in meeting its commitment to a 
90-percent reduction in the assembly defect rate because lt was not tracking lts progress using the 
correct data for SDU or LR1P lotdeliverïes. In accordance with cünflicting Information contaïned in 
the quality management plan, Lockheed Martin was tracking the number of nonconforming 
materia! records rather than the number of assembly defects, Lockheed Martin demonstrated a 23- 
pereent reduction in nonconforming material records compared to the 60-percent defect rate 
rediictioiT that shoüid have been realized after LRIP lot 5, in addition, |PO and DCMA did not ensure 
LhaL corrcclive nclions for product noncönfomnances were adcqualely implemented to reducc the 
assembly defecL rate. At the time of uur evaluaLion P we were net aware of any ïnttlaüves or val ue 
engineering change proposals speriJkally aimed at the reduction of the assembly defect rate in 
accordance with the program quality management pian goals, 

F-35 JPO Response: Disagree 

The program quality 90% objective the DODIG references was a ratio of Scrap Rework 
and Repair (SRR) to production labor hours (hence a 'rate 1 ). This is a normalization 
technique to assess Ie veis of SRR relativa to volume of work in a given area. This 
normalization is a valuable rnetric; however r it is not the only me trio. In the current case 
of F-35 the production line which is stil! experiencing significant production labor hour 
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reductions as a re suft of pfogressïng down predictable tearning, the ratio of dédining 
SRR hours to declining production labor hours has remained relativety steady. The 
F-35 JPO, DCMA and LM have worked to onderstand this phenomenon and anaiyzed 
other metrics to evaluate the program s progress toward reductions necessary to reach 
high production rates by increasing efficiency and flow. 

• Factory SRR hours have decreased 65% since the first production lot Supplier 
responsible SRR has been redueed 80% and the program is setting a goal to reduce 
the current fevel another 25% again this year 
» Quality Escapes to the fietd are another measure, which have decreased 75% since 
2011 


Recommendations 

D< We recommend that the F-3S Joint Program Office; 

1 Lnsure LhaL Lockheed Martin fmplemenls injtiaüues La reduce iheassemhly delecL ratecd Lhe 
product to meet plauned fUU rate productïon goals. 

F-35 JPO Response: Agree 

As presented, Lockheed Martin has initiated several initiatives to reduce 
nonoonformances of all types both within their fadlities and throughout their suppiy 
chains. The F-35 JPO is commifted to ensuring quality object]ves and goals are 
realized, therefore monitors progress cJosely with monthly and quarterly quality metrics 
reviews and Production Readiness Reviews Additionally F-35 JPO production 
operations and quafity subject matter experts attend Systems Engineering TechnicaJ 
Reviews to assure designs are produdble and encompass quality objectives. Additionai 
improvement incentives are being realized through the program s Bluepnnt for 
Affordability which pro vides a cost share between the contractor and government to 
fund the improvements, while not solely dedicated to quality and producibility 
im prove men ts the majority of projects achteve savings through defect reduction by 
föcusing on producibility. F-35 JSPQ is satisfied with the methodotogy LM is using to 
monitor SRR in fieu of a ratio to production hours. Additiona! metrics are being 
evaluated by F-35 JPO r DCMA and the contractors independently and jointiy to 
continue drïving discovery of nonconformances at their source (eliminate escapes) as 
well as facilitating correction. The mutually agreed to quality goals are part of the Joint 
Program Commitments, which the F-35 JPO may consider impJementing in contracts, 
though they are not currently. 

S.Coürdinate with DCMA to implementan effective rootcause analysisandcorrectiveactron 
process in order to meet assembly defect rate goals. 
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F-35 JPO Response: Partialfy Agree 

There are effective Corrective Action processes already in place within Lockheed Martin 
and the supply Chain. DCMA and F-35 JPO quality organizations focus heavily on 
continually improving these systems and process to assure cause analysis leads to true 
root cause and does not stop with direct cause. This includes areas outside production, 
induding work instructions, tooling, design, facilities, human resources and even within 
the government should govemment fumished equipment property or direction be a 
contributor Lockheed Martin has also impiemented intemal corrective action processes 
for non-shop floor continual improvement opportunities White F-35 JPO agrees 
effect ive root cause and corrective action are essential to im prove ment, beyond 
continuous high priority focus through Corrective Action Board meetings, metrics 
re views, DCMA issuance of requests for corrective action no additional changes are 
necessary at this time. 


Finding E 

Corrective Action Request Escalation Was Inadequate 

DCMA Fort Worth was not escalating unresolved CARs to the next higher level as required by the 
applicabk? DCMA instructiou. lf DCMA does not escalate CARs as required, defidencies may 
continue to occurand DCMA and ]PO leadership may not beaware ofproblems untll they adversely 
impact the cost r Schedule. and performance of the program. 

Discussion 

DCMA was not folio wing Hs policy to escataie CARs Ui the next higher level when repetïlive Level ll 
CARs have been issüed, when CAPs were rejected and not resubmitted wilhin the IfLday 
requiremenL or when the corrective aetïon vnlidation is rejected, We identified mstances when 
multiple CARs were docmnented for the same violation, but none were escalated to the ncxt level. 
We also noted instances when CAPs were rejected and were not resubinitted within the lÖ-day 
requïremenC but dld not result in DCMA escalating the CAR to the next level. 

DCMA Response: Agree 

DCMA agrees we did not execute to DCMA instruction #1201 as mterpreted by the DoD 
!G team. We agree that DCMA instruction 1201 requires clarification lt is currently 
under re Vision to address clarification to the CAR elevation process. 

DCMA believes this risk is mitigated based on a comprehensive program quality 
strategy between DCMA, F-35 JPO, and LM. There is a monthly rnetncs cadence in 
which the DCMA Quality organization provides independent CAR analysis The data 
presented in these meetings provides insight of non-comptiances identified through 
DCMA and LM Quality surveillance activities. 


Recommendations 
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E. Wc re co mm end that the Defcnse Contract Management Agency: 

1. Review aU unresolved CARs and escalate those that meet the criteria established in DCMA 
poli ei es and instructions. 

DCMA Response: Agree 

DCMA agrees we did not execute to DCMA Instruction #1201 as interpreted by the DoD 
IG team. We agree that DCMA instruction 1201 requires clanfication. It is currently 
under revision to address clarificatïon to the CAR elevation process. DCMA 
Headquarters wili provide an updated policy with an estimated compietion date of eariy 
2015. Upon poficy update approval, DCMA LMFW wilf re view all open CARS lo 
determine if any warrant elevation to the next level as recommended 

2, Assess all CARs that were not properly elevated and assess any impact on the product 

DCMA Response: Agree 

DCMA LMFW wiil assess alE open CARs based on new policy guidance fo identify, 
amalyze, and assess any impacts to F-35 product 


Fi ruling F 

Software Quality Management Was Insufficiënt 

[PO and Lockheed Martin were not ensuring software quality management pmcesscs such as 
contigu ra ti on management failure reportingand corrects ve action, nonconformancc root cause and 
analysis, and training were performed stifiïciently to prevent delayed mission capability, software 
rework, and possi bly additkmal program costs. In addition. un disco ve red software defects could 
result In performance degradation or even catastrophie fatlure of aircraft, subsequenüy resulting in 
loss of aircraft and death. 

Discussion 

Lockheed MorLirTs software development methodology did not facilitaLe the performance of 
physical and functional contiguration audits. The majority of new r aircraft capability is based on 
additional software bniids: therefore, it Is necessary to perform functional and physical 
configuration audits. These types of audits ensure that CSCls exist and are configured, tested, and 
doemnented in accordance with software specifications and requi reine nts to mïtïgate aircraft 
performance and reliability risks. In addition, deficiencies in Lockheed Martin's software 
development training program may lead to knowledge gaps which could resuit in software 
development process failures and software defects. Addiüonaily, the dassiflcation of the severity of 
software defici enctes idenlilles the level of verifïcation and correct!ve action required. Lockheed 
Martin's fatlure to ensure software subcontractors, such as Northrop Grumman, were categorhsing 
software defieiencies in accordance with the F-35 Air System Software Devclopment Plan does not 
provide confidenee in dolivered software. Allowing open deficiencies, especially those whosc 
severftles are not characterized and understood, eau result in performance degradation or 
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catastrophic failure du ring operation. JPO need$ to ensure that rigorous software processes exist at 
Lockheed Martin, 

F-35 JSPO Response: Disagree 

The F-35 JPO considers F-35 software quality management is sufficiënt, specifïcally the 
airvehicle software quatity referenced in the draft DOD IG audit report. The F-35 JPO is 
confident current air vehicle software configurafion management, failure reportimg and 
corrective action, root cause analysis, and training are sufficiënt and adequate to 
support program mission capability and cost objectives. The F-35 JPO is confident in 
the current air vehicle software, systems engineering, and airworthiness certification 
processes will ensure no undiscovered software defects will be delivered with the air 
vehicle that coutd result in performance degradation or even catastrophic failure of 
aircraft. 

Specific DOD IG Findinq Responses: 

1 DOD IG Finding Lockheed Martin's software development methodology did not 
facititate the performance of physical and functional configurafion audits. The 
majority of new aircraft capability is based on additional software builds; 
therefore, it is necessary to perform functional and physical configuration audits. 
These types of audits ensure that CSCIs exist and are configured, tested, and 
documented in accordanee with software specifications and requirements to 
mitigate aircraft performance and reliability risks. 

F-35 JPO Response: In accorda nee with the F-35 program Software 
Development Plan and Configuration Management Plan, physical and functional 
configuration audits of the type referenced by the DOD IG report are conducted 
against the air vehicle Software Data Load T which is a compilation of vehicle and 
mission systems components. The program is conducting FCA/PCAs in 
accordance with the F-35 CM Plan, 2YZA00002, on all Cis and CSCIs identified 
in the F-35 Cl List, 2YZA0094T approved by the F-35 JPO, This finding 
erroneously presumes that certain software items which are not identified in the 
Cl list are CSCIs and therefore must be subject to FCA/PCA requirements 
Mission System and Vehicle System computer programs are elements which 
contribute to the composition of the aircraft Software Data Load. It is the aircraft 
Software Data Load which is identified in the F-35 Cl List as a CSCI, and it is this 
software product on which FCA/PCA activity will be conducted. This practice is 
consistent with software development practices on all weapon systems 
developed at the Fort Worth facility, including the F-16 and F-22, in accordance 
with the DoD required Single Process Initiative for the Fort Worth Facility. The 
F-35 program has a very rigorous and structured verification and validation 
process for software enabled systems capabilities, including the Identification and 
management of all software components that contribute to safety critical 
functions. In addition, F-35 air vehicle software undergoes extensive independent 
reviews by external airworthiness agencies from the United States Air Force and 
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Navy in the context of MIL-HDBK-516, as well as the Ministries of Defense of the 
United Kingdom and Australia. These agencies have ïndependently confirmed 
the adequacy of the physical configuration and functional ability of the air vehicle 
software for safe operation in the delivered operational envelope The F-35 
program does nat infend to modify existing functional and physical configuration 
audit practices for F-35 air vehicle software based on this finding. 

2. DoD IG Finding: Deficiencies in Lockheed Martin's software development training 
program may lead to knowledge gaps which could result in software 
development process failures and software defects. 

F-35 JPO Response: Lockheed Martin has a structured and 
accountable training program for all software engineers in the LM 
aeronauttcal diVision, documented in PM-4002 {Software Engineering 
Training). In addition, the LM F-35 program has a structured and 
accountable training program for F-35 spedfic software development 
environments and tools. Training history for both LM Aero and F-35 
specific training were provided to the IG team, as well as Beaming 
plans for specific classes of software engineering disciplines. Air 
vehicle software engineers on the F-35 program comphse a body of 
world class talent - they are industry experts in the development of 
hard real time embedded flight and mission critical software, and have 
in most cases decades of experience in vehicle and mission systems 
software development on other 4 lh and 5 th gene ratio n fighter programs. 

They are sought after by industry to provide instruction to other 
engineers in their discipline areas, and werk directly with the 
developers of the software tools and environments they are using to 
incorporate additional enhancements and features to increase 
productivity and efficiency. The F-35 JPO does not intend to pursue 
any additional training initiatives for the LM software development team 
based on this finding. 

3. DoD IG Finding: The classification of the severity of software deficiencies 
identifies the level of verification and corrective action re q ui red. Lockheed 
Martin's failure fo ensure software subcontractors, such as Northrop Grumman, 
were categorizing software deficiencies in accordance with the F-35 Air System 
Software Development Plan does not provide confidence in delivered software. 
Allowing open deficiencies, especialiy those whose severities are not 
characterized and understood, can result in performance d eg ra dat ion or 
catastrophic failure during operation. 

F-35 JPO Response: System and software deficiency severity classification and 
tracking is managed and maintained at the air vehicle System level, in 
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accordance with F-35 program systems engineering, software engineering, and 
system verificatie n plans The DoD IG team assumed subcontractors wou ld 
independently classify and track their anomalies using the same criteria as the air 
vehicle teams, which is not the defined process or practice Subcontractor 
deficiencres are reported against the specification requirements fevied to the 
subcontractors, but carmot be categorized by the subcontractor team ïn terms of 
air vehïcle severity, as only the air vehicle systems engineering teams in the 
vehicle systems and mission systems functional areas have the required breadth 
and scope of system functtonality and safety criticality to adequately classify the 
severity of component anomalies in the context of air vehicle operations The 
F-35 JPO reviewed the specific concerns highlighted by the DoD IG with 
Northrop Grumman faüure reporting, and found the reporting practice consistent 
with the documented air vehicle software, sysfems engineering, and air vehicle 
verification plan. There are no open deficiencies delivered with the air vehicle 
that are not characterized by hazard severity and understood within the context 
of mission and safety criticality, and which have not been adjudicated with the 
F-35 JPO, airworthiness authorities, and operational users. The F-35 JPO does 
not plan to modify the existing software, systems engineering, and system 
verification process or practices based on thïs finding. 


Recommendation 

F. We recommend Lhat the F-35 joint Program Office work with Lockheed Martin to ensure 
software quality management systems are improved; metrics shouid be reported on a periodic 
basis ffor example, monthly} to evaluate process improvemenL 

F-35 JPO Response: Agree 

The F-35 JPO has been and will continue to work with Lockheed Martin to ensure the 
adequacy and ïmprovement of software quality management processes and practices 
Metrics reflecting software quality and quality improvement objectives have been 
developed and are being reported and assessed at both the vehicle systems and 
mission systems team levels on a monthly basis, as wed as at the au vehicle level and 
production quality on a monthly basis 


Finding G 

Subcontract Management Procedures Were Still 
Inadequate 

|PD, DCMA, and Lockheed Martin were not ensiuing that contract requireinents were adequately 
flowed down, contract deliverables were evaluated, and minor nonconformancos were approved by 
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F-35 Joint Program Office and Defense 
Contract Management Agency Response to 
Findings/Recommendations (cont'd) 
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the proper suthority. Lack ofsulTirieriLsubcantract miinagemenL gcLivilies may resull in Ihe 
acceptance of products tliat do not meet intended performance and relïabilïty requirements. 

Discussion 

DCMA and Lockheed Martin were not ensuring that the purchase orders identlfied the speelde 
revisions of the technicai requirement documentatJon necessaiy for the production, testing. and 
inspectiën ofeach S tem being purchased and! changes to Leehnical requirements were cofitracLuaily 
documented, They also dïd nolensure Üiattoniractually requlred suppïïer-geïierated data items 
were dehvered by subcontractors wben required, We found no ci/idence oTany appi opriate actions 
taken vvhen subcontractors did not comply with these requirements. in additiom [PO and Lockheed 
Martin were not ensuring that dis position and approval ol minor nonconformances we re spproved 
only by DCMA P who lias the delegated authority for approval of minor nonconformances. 

Recommendations 

G, 1. We recommend that the 1--35 Joint Program Office ensure that all minor nonconfoimances are 
evaluatcd and approved oniy by PCMA, 

F-35 JPO Response; Partially Agree 

F-35 JPO will work with DCMA to assure compliance with regulation and drrection. As 
part of an existing effort to validate requirements flow down accuracy, minor 
nonoonformance spproval will be incEuded. Scrap and rework-to-design-definition (so 
cailecf re work to blue-print) dispositions ultimately result in conform in g product being 
presented to the government for acceptance and alleviate the requirement of further 
DCMA review. DCMA has and witl continue to intercede in repetitive scrap and rework 
In the case of repair or use-as-is disposition, DCMA reviews these in accordance with 
reguiation and instruction as it results in nonconforming product being tendered for 
acceptance. The F-35 JPO will work with DCMA to assure nonconformance 
evaluations are properly reviewed in accordance with requirements and action plans put 
in place by March 31,2015. 

G.2. We recommend Lhat the Defense Contract Management Agency ensure that Lockheed Martin 
flows down the appropriate technical requirements to its subcontractors and receivesand 
evaluates contract del ivera bles with in the requ tred time frames, 

DCMA Response: Agree 

DCMA LM wil! assess current functionai surveillance strategies to determine 
system/process surveillance gaps regarding Subcontract Management Procedures 
DCMA LM surveillance strategies will be revised as necessary to ensure LM processes 
are adequate to provide appropriate flow down of requirements and execution of those 
requirements are achieved by their subcontractors. 
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Lockheed Martin Letter to F-35 Joint Program Office 
on Quality Improvements 


LOCKHEED «Aft 




Dr Alexander Eksrr 

Vice President, Quality & Mission Success 
Lockheed Martin Aeronautics 


DATE: 

TO: 

ATTN: 

SUBJECT: 

REFERËNCË: 


13 January 2015 

F-35 Joint Strike Fighter Program Office 



Update and Follow Up to 2014 Department of Defense Inspector General 
Quality System Review at Lockheed Martin Aeronautics 

Dïscussion with Department of Defense Inspector General; Please 
Consider for Inclusion with Response to F-35 Ughtning II Program Quality 
Assurance and Corrective Action Report 


To the F-35 Joint Strike Fighter Program Office: 

As noted du ring the latest IG review in September, much progress is being made at Lockheed 
Martin Aeronautics on the F-35 program. The company contmues to work the issues while 
launching initiatives to ensure an enduring focus on process excellence and zero defects. 
Following the recent review in 2014, the company quickly took action to address the concerns 
identified by the audit team. Lockheed Martin Aeronautics in partnership with DCMA and the 
JPQ do cu menled observations made by the audit team with intern al Correct Eve Action Requests 
(iCARs). A Forma! Corrective Action Plan (GAP) was developed for each iCAR allowing Subject 
Matter Experts (SMEs)* in collaboration with local DCMA and JPQ representaties, to drive robust 
rooFcause anaEysis, close systemic gaps, and jointly vaiidate effectiveness for each fïnding. In 
addïtion to these corrective act! ons, the JPQ continues to evolve and strengthen management 
oversight to ensure our momentum continues. Added oversight methods tnclude multiple quality 
system re views, metrïcs reviews, joint audlts and other regulariy scheduled events that are 
planned one-year in ad va nee and tracked monthly. 

The JPO and Lockheed Martin Aeronautics have ieveraged the IG reviews to improve the 
company Quality Management System with several changes including the followlng: 

1. Review formats, content, and accountabiiities were reshaped while ensuring broader 
i n clusi o n of DC MA a n d othe r prog ra m matic sta keh olders, I n 2014 a lone over 20 
enhanced quality reviews were heid between DCMA, JPO, Lockheed Martin Aeronautics, 
and key suppliers/ co-producers. Many of these events will continue to be conducted 
weekly and monthly to drive improvements while collecting the real-time status of quality 
performance and addressing program quality challenges. 

2. Th e G lobal Qu a lity Co u ncil (G QC) was created to s u pp o rt supply c hai n i m prove merite a nd 
share best practices between industry partners. The GQC is chafred by the Lockheed 
Martin Aeronautics Vice President of Q&MS and represents the top seventeen mission 
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critlcal major suppliers and co-producers for F-35. The GQC is actively engaged in 
improvement and quality transformations across the suppiy Chain, During 2014, the 
councii facilitated joint engagements to address common problem sub-tier suppliers, offer 
training exchanges between GQC members, improved response times to address 
challenges between companies, and execute resource exchanges to address technical 
and quality issues, 

3. The Aeronautics Quality Transformation (AQT) was established to further Lockheed 
M arti n Aeronautics’ focus o n quai ity i m proveme nt C ha m pio ned by the bu s tness 
president and sponsored by senior executives, the focus of the AQT is on cultivatlng 
systematic accountability to quality across the company and value chain. AQT 
challenges each section of the company to identify areas of improvement and investments 
that will drive out defects. Many of the improvements noted during the DoD IG au dit and 
positive changes at Lockheed Martin Aeronautics are directly tïed to the corporate 
sponsored AQT efforts and projects, 

4. Fort WortlYs FOD Prevention Center was erected as a state-of-the-art, hands-on learning 
clinic for FOD detection and prevention, The center has provided an excellent venue for 
training induding FOD detection and retrieval, enhanced specialized equipment training, 
and a new sïmulatïon trainer, The annual compliance training "Fundamentals of FOD 
Prevention 1 2 3 4 5 6 ' has been refreshed to deliver improved learning. Process compliance 
assessments are performed by the FQD prevention team across all factory build areas. 
Alt hou gh FOD continues to be a challenge for Lockheed Martin Aeronautics and industry, 
real improvements have been established and are yielding resuits across the enterprïse, 

5. Learning and deveiopment of human capital continue to be key investments toward 
continuous improvement Investments induding the Miss ton Success Deveiopment 
Training initiative (MSDTi), training and certification by extemal institutes, and the quatity 
engineering transformatïon, were instituted to strengthen employee understanding and 
abïlity in Quality Management and Science across Lockheed Martin Aeronautics, MSTDi 
is a twelve month training curriculum applying the principles of quality. More than 200 
students from various functions at the company and its suppliers have completed the 
MSTDi program and 40 percent of quatity engineers are certified by the American Society 
fo r Qu ai ity. 100 pe ree nt (154) of Q & M S leade rsh i p an d sa lari e d em p loyee s have 
completed AS910Q training, and 100 percent of ïnspectors have completed a “Q101” 
training course. 

The focus areas mentioned are just a few examples of the work underway. These efforts are 
yielding resuits with a more mature Quality Management System, product quality improvements, 
and program momentum. Examples of the key indicators of quality improvement include the 
following; 

1. A 50 percent reduction in defects from the top 22 F-35 suppliers during 2014; 

2. 80 percent reduction in supplier Scrap, Rework, and Repair (SRR) since LRtP 1; 

3. Factory SRR hours have decreased 65 percent between LRIP 1 and LRIP 6; 

4. Production-caused escapes to the field are down 75 percent since 2011; 

5. Zero FOD defects at delivery in 2014; 

6. Flïght test performance continues to accelerate and validate the capability of the platform. 
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Difficult early program technical challenges are being resoived in partnership with Lockheed 
Martin Aeronautics, the JPO, and program suppliers. Contracting challenges are being dillgently 
worked and resoived, Affordability continu es to be a focus area with multiple efforts resuiting in 
reduced oosts as evident in LRIP over LRIP price reductions. 

The Lockheed Martin Aeronautics continuous improvement strategy is being realized ïn a way 
that is benefiting the F-35 program as well as the other platforms and sites that make up the 
business, The assessments by the DoD IG havo further validated those ïmprovements and 
areas of needed focus. 
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Acronyms and Abbreviations 


APUC Average Procurement Unit Cost 
AS Aerospace Standard 
CAP Corrective Action Plan 
CAR Corrective Action Request 
CCDD Capability Description Document 
CSCI Computer Software Configuration Item 
CSI Critical Safety Item 
CTOL Conventional Takeoff and Landing 
CV Carrier-Suitable Variant 
DCMA Defense Contract Management Agency 
DFARS Defense Federal Acquisition Regulation Supplement 

DMSMS Diminishing Manufacturing Sources and Material Shortages 
FCA Functional Configuration Audit 
FOD Foreign Object Debris 

FRACAS Failure Reporting, Analysis, and Corrective Action System 
HWCI Hardware Configuration Item 
ICA Integrated Corrective Action 

ICNI Integrated Communications, Navigation, and Identification 
IG Inspector General 
JCS JSF Contract Specification 
JPO Joint Program Office 
JSF Joint Strike Fighter 
LRIP Low-Rate Initial Production 
OFI Opportunity for Improvement 
PAUC Program Acquisition Unit Cost 
PCA Physical Configuration Audit 
PEO Program Executive Officer 
SDD System Development and Demonstration 
SDP Software Development Plan 
SPAR Software Problem Anomaly Report 
STOVL Short Takeoff and Vertical Landing 
UTAS United Technologies Corporation, Aerospace Systems 
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Whistleblower Protection 

U.S. Department of Defense 


The Whistleblower Protection Enhancement Act of 2012 requires 
the Inspector General to designate a Whistleblower Protection 
Ombudsman to educate agency employees about prohibitions 
on retaliation, and rights and remedies against retaliation for 
protected disclosures. The designated ombudsman is the DoD Hotline 
Director. For more information onyour rights and remedies against 
retaliation, visit www.dodig.mil/programs/whistleblower. 


For more information about DoD IG 
reports or activities, please contact us: 

Congressional Liaison 

congressional@dodig.mil; 703.604.8324 

Media Contact 

public.affairs@dodig.mil; 703.604.8324 

Monthly Update 

dodigconnect-request@listserve.com 

Reports Mailing List 

dodig_report@listserve.com 

Twitter 

twitter.com/DoD_IG 

DoD Hotline 

dodig.mil/hotline 


DEPARTMENT OF DEFENSE | INSPECTO 

4800 Mark Center Drive 
Alexandria, VA 22350-1500 
www.dodig.mil 

Defense Hotline 1.800.424.9098 













